pcDuino9 AI視覺邊緣計(jì)算開發(fā)套件開發(fā)使用分享

?

pcDuino9 AI視覺邊緣計(jì)算開發(fā)套件是一款針對(duì)圖像分析的開發(fā)套件，套件包含采用hi3516D方案的普通IPC和pcDuino9嵌入式AI視覺邊緣計(jì)算模塊。套件主要是通過pcDuino9嵌入式AI視覺邊緣計(jì)算模塊，來處理從hi3516D上獲取的視頻流，并對(duì)視頻流進(jìn)行一系列的分析處理。該套件為開發(fā)者搭建了基礎(chǔ)開發(fā)環(huán)境，提供了由pcDuino9從hi3516D上獲取的視頻幀，開發(fā)者可在幀圖上自由的實(shí)現(xiàn)各種圖片開發(fā)，例如人臉檢測、車牌檢測、物體檢測等AI圖像分析處理功能。同時(shí)提供開放的API接口，開發(fā)者可以把在pcDuino9上處理完成的信息回傳到hi3516D上，并可在hi3516D上對(duì)檢測到的圖片進(jìn)行刪選，然后通過FTP輸出。

?

基本框架

?

已提供環(huán)境：

1.?pcDuino9 從hi3516D獲取視頻幀

2.?把開發(fā)者分析處理后的信息回傳到hi3516D上

3.?在hi3516D上對(duì)分析處理的圖片進(jìn)行質(zhì)量篩選，選擇質(zhì)量優(yōu)的圖片通過FTP輸出

開發(fā)舉例：

1.?在pcDuino9上對(duì)幀圖進(jìn)行各種開發(fā)，如：人臉檢測、車牌檢測、物體檢測等AI圖像分析處理。

2.?開發(fā)者可對(duì)通過FTP輸出的分析檢測的圖片進(jìn)行其他應(yīng)用上的使用

3.?……… ??

?

開放SDK資料

1.?如何獲取hi3516D視頻幀？

HI3516將BT1120數(shù)據(jù)傳輸?shù)絇CDUINO 9。PCDUINO 9通過LT8918將BT1120數(shù)據(jù)轉(zhuǎn)換成MIPI信號(hào)，然后接收。目前支持640x480 60fps幀數(shù)據(jù)傳輸。我們封裝了協(xié)議層，使其類似于opencv樣式。在讀取幀數(shù)據(jù)時(shí)，我們將返回幀號(hào)，該幀號(hào)由3516傳輸，是用于數(shù)據(jù)交互的同步幀號(hào)。

#include

#include

#include "C4L2.hpp"

#include

using namespace std;

using namespace cv;

int main(int argc,char *argv[])

{

string dev = "/dev/video2";

C4L2Capture *cap = new C4L2Capture;

int ret = cap->initialzer(dev);

if(ret < 0)

{

cout << "C4L2Capture initialze failed... \n";

return -1;

}

Mat frame;

while(true)

{

unsigned int syn = 0;

cap->read(frame, syn);

if(frame.empty())

{

cout << "this frame is empty ...\n";

?

break;

}

cv::imshow("DeepCam LLC",frame);

cv::waitKey(10);

}

cap->destroy();

delete cap;

return 0;

}

2.?如何優(yōu)化網(wǎng)絡(luò)并運(yùn)行物體檢測？

PcDuino9有一個(gè)四核Cortex-A17 1.8GHz的CPU和一個(gè)ARM MALI-T764的GPU。如果您想在CPU上運(yùn)行網(wǎng)絡(luò)，可以參考NCNN；如果您想在GPU上運(yùn)行網(wǎng)絡(luò)，可以參考MACE。

CPU Benchmark:

?

物體檢測：

#include

#include

#include

#include

#include

#include "net.h"

class Noop : public ncnn::Layer {};

DEFINE_LAYER_CREATOR(Noop)

struct Object

{

cv::Rect_<float> rect;

int label;

float prob;

};

static int detect_mobilenetv2(const cv::Mat& bgr, std::vector<Object>& objects)

{

ncnn::Net mobilenetv2;

mobilenetv2.register_custom_layer("Silence", Noop_layer_creator);

// original pretrained model from https://github.com/chuanqi305/MobileNetv2-SSDLite

// https://github.com/chuanqi305/MobileNetv2-

SSDLite/blob/master/ssdlite/voc/deploy.prototxt

mobilenetv2.load_param("mobilenetv2_ssdlite_voc.param");

mobilenetv2.load_model("mobilenetv2_ssdlite_voc.bin");

const int target_size = 300;

int img_w = bgr.cols;

int img_h = bgr.rows;

ncnn::Mat in = ncnn::Mat::from_pixels_resize(bgr.data, ncnn::Mat::PIXEL_BGR,

bgr.cols, bgr.rows, target_size, target_size);

const float mean_vals[3] = {127.5f, 127.5f, 127.5f};

const float norm_vals[3] = {1.0/127.5,1.0/127.5,1.0/127.5};

in.substract_mean_normalize(mean_vals, norm_vals);

ncnn::Extractor ex = mobilenetv2.create_extractor();

ex.set_light_mode(true);

ex.set_num_threads(4);

ex.input("data", in);

ncnn::Mat out;

ex.extract("detection_out",out);

// printf("%d %d %d\n", out.w, out.h, out.c);

objects.clear();

for (int i=0; i<out.h; i++)

{

const float* values = out.row(i);

Object object;

object.label = values[0];

object.prob = values[1];

object.rect.x = values[2] * img_w;

object.rect.y = values[3] * img_h;

object.rect.width = values[4] * img_w - object.rect.x;

object.rect.height = values[5] * img_h - object.rect.y;

objects.push_back(object);

}

return 0;

}

static void draw_objects(const cv::Mat& bgr, const std::vector<Object>& objects)

{

static const char* class_names[] = {"background",

"aeroplane", "bicycle", "bird", "boat",

"bottle", "bus", "car", "cat", "chair",

"cow", "diningtable", "dog", "horse",

"motorbike", "person", "pottedplant",

"sheep", "sofa", "train", "tvmonitor"};

cv::Mat image = bgr.clone();

for (size_t i = 0; i < objects.size(); i++)

{

const Object& obj = objects[i];

fprintf(stderr, "%d = %.5f at %.2f %.2f %.2f x %.2f\n", obj.label, obj.prob,

obj.rect.x, obj.rect.y, obj.rect.width, obj.rect.height);

cv::rectangle(image, obj.rect, cv::Scalar(255, 0, 0));

char text[256];

sprintf(text, "%s %.1f%%", class_names[obj.label], obj.prob * 100);

int baseLine = 0;

cv::Size label_size = cv::getTextSize(text, cv::FONT_HERSHEY_SIMPLEX, 0.5, 1,

&baseLine);

int x = obj.rect.x;

int y = obj.rect.y - label_size.height - baseLine;

if (y < 0)

y = 0;

if (x + label_size.width > image.cols)

x = image.cols - label_size.width;

cv::rectangle(image, cv::Rect(cv::Point(x, y),

cv::Size(label_size.width, label_size.height +

baseLine)),

cv::Scalar(255, 255, 255), CV_FILLED);

cv::putText(image, text, cv::Point(x, y + label_size.height),

cv::FONT_HERSHEY_SIMPLEX, 0.5, cv::Scalar(0, 0, 0));

}

cv::imshow("image", image);

cv::waitKey(0);

}

int main(int argc, char** argv)

{

if (argc != 2)

{

fprintf(stderr, "Usage: %s [imagepath]\n", argv[0]);

return -1;

}

const char* imagepath = argv[1];

cv::Mat m = cv::imread(imagepath, CV_LOAD_IMAGE_COLOR);

if (m.empty())

{

fprintf(stderr, "cv::imread %s failed\n", imagepath);

return -1;

}

std::vector<Object> objects;

detect_mobilenetv2(m, objects);

draw_objects(m, objects);

return 0;

}

3.?如何和IPC通信？

我們提供一個(gè)開放式API接口。開發(fā)人員可以將pcDuino9上處理的信息返回到HI3516D，在HI3516D上對(duì)檢測到的圖像進(jìn)行優(yōu)化選擇，然后通過ftp輸出。

#include

#include "json/json.h"

#include "spi_slave.hpp"

using namespace std;

struct DetectInfo

{

int ID; //tracker ID

int x; // bbox left top x

int y; // bbox left top y

int w; // bbox width

int h; // bbox height;

int syn; // sync frame count

int quality; // object quality (0 -- 100)

int confidence; // object confidence (0 -- 100)

};

void detect_out(std::vector<DetectInfo> &detectBoxs, std::string &message)

{

Json::Value root;

Json::Value faces;

Json::FastWriter writer;

for(int i = 0; i < detectBoxs.size(); i++)

{

Json::Value face;

face["x"] = detectBoxs[i].x;

face["y"] = detectBoxs[i].y;

face["w"] = detectBoxs[i].w;

face["h"] = detectBoxs[i].h;

face["id"] = detectBoxs[i].ID;

face["s"] = detectBoxs[i].ID;

face["q"] = detectBoxs[i].quality;

face["c"] = detectBoxs[i].confidence;

faces.append(face);

}

root["faces"] = faces;

message = writer.write(root);

}

int main(int argc, char *argv[])

{

unsigned char SOF;

spi_slave *rk3288_spi = new spi_slave;

if(rk3288_spi->init() < 0)

{

printf("spi test spi_init error.\n");

return -1;

}

unsigned char *r_buf = new unsigned char[MAX_DATA_LENGTH];

unsigned char *w_buf = new unsigned char[MAX_DATA_LENGTH];

while (true)

{

std::string message;

std::vector<DetectInfo> detectBoxs;

//Do object detect get message

detect_out(detectBoxs, message);

SOF = 0;

memset(r_buf, 0, MAX_DATA_LENGTH);

strcpy((char *)w_buf,message.c_str());

int ret = rk3288_spi->spi_slave_xfer(r_buf,MAX_DATA_LENGTH, w_buf,

MAX_DATA_LENGTH, &SOF);

if(ret > 0 )

{

memset(w_buf, 0, MAX_DATA_LENGTH);

if(SPI_M_TX == SOF)

{

std::cout << "recv : " << r_buf << std::endl;

usleep(30000);

}

}

else

{

std::cout << "get message failed!\n";

break;

}

}

return 0;

}

?

硬件介紹

pcDuino9嵌入式AI視覺邊緣計(jì)算模塊介紹

接口圖

?

引腳圖

?

?

原文鏈接：

http://linksprite.com/wiki/index.php?title=PcDuino9_AI_visual_edge_computing_development_kit

?