fall-axera

This version of fall-axera has been converted to run on the Axera NPU using w8a16 quantization. It is trained with modified yolov7-pose model to detect bbox and 14 keypoints of human, and to determine whether a fall behavior is likely to occur.

Supported Classes

This model is trained to detect the following classes:

1. normal
2. fall

Supported keypoints

This model is trained to detect the following 14 keypoints:

"keypoints": { 0: "right shoulder", 1: "right elbow", 2: "right wrist", 3: "left shoulder", 4: "left elbow", 5: "left wrist", 6: "right hip", 7: "right knee", 8: "right ankle", 9: "left hip", 10: "left knee", 11: "left ankle", 12: "head tops", 13: "upper neck" }"

Compatible with Pulsar2 version: 5.2.

Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through:

• The repo of AXera Platform, where you can get the detailed guide.
• Pulsar2 Link, How to Convert ONNX to axmodel

Support Platform

https://docs.m5stack.com/zh_CN/ai_hardware/AI_Pyramid-Pro

• AX650N/AX8850
  • M4N-Dock(爱芯派Pro)
  • AI Pyramid
  • M.2 Accelerator card

How to use

Download all files from this repository to the device.

python env requirement

pyaxengine

https://github.com/AXERA-TECH/pyaxengine

wget https://github.com/AXERA-TECH/pyaxengine/releases/download/0.1.3.rc2/axengine-0.1.3-py3-none-any.whl
pip install axengine-0.1.3-py3-none-any.whl

Inference with AX650 Host, such as M4N-Dock(爱芯派Pro)

Input image:

run

python3 axmodel_infer_fall.py

root@ax650:~/fall-axera# python3 axmodel_infer_fall.py
[INFO] Available providers:  ['AxEngineExecutionProvider', 'AXCLRTExecutionProvider']
[INFO] Using provider: AxEngineExecutionProvider
[INFO] Chip type: ChipType.MC50
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Engine version: 2.12.0s
[INFO] Model type: 2 (triple core)
[INFO] Compiler version: 5.2 eccb31f5
class: fall left:281 top:396 right:734 bottom:629 conf: 74%
Result saved to axmodel_res.jpg

Output image:

Extra

This example only shows the model's predicted bounding boxes and keypoints. You can further assist in determining human falls based on the physical information of the boxes and keypoints, or by adding tracking and action recognition models like st-gcn. From the experiments I have conducted, factors such as occlusion, direction of falling, camera angle, and even the scene (such as on the bed or on the floor) can affect the results of fall detection.