Difference between revisions of "Dev:AI/CVAT"

From C3LearningLabs
(→‎Log)
(→‎Install)
(2 intermediate revisions by the same user not shown)
Line 288: Line 288:
 
:/mnt/cvat/baldr/concopy
 
:/mnt/cvat/baldr/concopy
  
 +
 +
=== Local Browser ===
 +
 +
sudo apt install chromium-browser
 +
 +
 +
ssh -Y -l regogr -p 23122 10.112.46.3
 +
chromium-browser
 +
 +
=== Good to have ===
 +
sudo apt install nmap
 +
nmap localhost
 +
nmap 192.168.122.31
 +
nmap 192.168.122.31 -p8000-9000
 +
 +
wget 192.168.122.31:8080
 +
docker-compose down
 +
nano docker-compose.yml
 +
docker-compose up -d
  
 
=== ConCopy ===
 
=== ConCopy ===
Line 319: Line 338:
 
cvat_logs:/mnt/cvat/baldr/django/logs<br>
 
cvat_logs:/mnt/cvat/baldr/django/logs<br>
 
cvat_models:/mnt/cvat/baldr/django/models<br>
 
cvat_models:/mnt/cvat/baldr/django/models<br>
 +
 +
 +
Fredrik funkar
 +
<source>
 +
#
 +
# Copyright (C) 2018 Intel Corporation
 +
#
 +
# SPDX-License-Identifier: MIT
 +
#
 +
version: "2.3"
 +
 +
services:
 +
  cvat_db:
 +
    container_name: cvat_db
 +
    image: postgres:10.3-alpine
 +
    networks:
 +
      default:
 +
        aliases:
 +
          - db
 +
    restart: always
 +
    environment:
 +
      POSTGRES_USER: root
 +
      POSTGRES_DB: cvat
 +
    volumes:
 +
      - cvat_db:/var/lib/postgresql/data
 +
 +
  cvat_redis:
 +
    container_name: cvat_redis
 +
    image: redis:4.0.5-alpine
 +
    networks:
 +
      default:
 +
        aliases:
 +
          - redis
 +
    restart: always
 +
 +
  cvat:
 +
    container_name: cvat
 +
    image: cvat
 +
    restart: always
 +
    depends_on:
 +
      - cvat_redis
 +
      - cvat_db
 +
    ports:
 +
      - "8080:8080"
 +
    build:
 +
      context: .
 +
      args:
 +
        http_proxy:
 +
        https_proxy:
 +
        no_proxy:
 +
        socks_proxy:
 +
        TF_ANNOTATION: "no"
 +
        AUTO_SEGMENTATION: "no"
 +
        USER: "django"
 +
        DJANGO_CONFIGURATION: "production"
 +
        WITH_TESTS: "no"
 +
        TZ: "Etc/UTC"
 +
        OPENVINO_TOOLKIT: "no"
 +
    environment:
 +
      DJANGO_MODWSGI_EXTRA_ARGS: ""
 +
      UI_PORT: 7080
 +
      ALLOWED_HOSTS: "*"
 +
 +
 +
    volumes:
 +
      - cvat_data:/home/django/data
 +
      - cvat_keys:/home/django/keys
 +
      - cvat_logs:/home/django/logs
 +
      - cvat_models:/home/django/models
 +
 +
  cvat_ui:
 +
    container_name: cvat_ui
 +
    image: nginx
 +
    restart: always
 +
    build:
 +
      context: .
 +
      args:
 +
        http_proxy:
 +
        https_proxy:
 +
        no_proxy:
 +
        socks_proxy:
 +
      dockerfile: Dockerfile.ui
 +
    environment:
 +
      REACT_APP_API_PROTOCOL: http
 +
      REACT_APP_API_HOST: localhost
 +
      REACT_APP_API_PORT: 8080
 +
    networks:
 +
      default:
 +
        aliases:
 +
          - ui
 +
    depends_on:
 +
      - cvat
 +
    ports:
 +
      - "7080:80"
 +
 +
volumes:
 +
  cvat_db:
 +
  cvat_data:
 +
  cvat_keys:
 +
  cvat_logs:
 +
  cvat_models:
 +
 +
</source>
 +
 +
http://10.112.46.3:23188/admin/
 +
  
 
<source>
 
<source>

Revision as of 10:17, 3 December 2019

Links

https://github.com/opencv/cvat

https://software.intel.com/en-us/articles/computer-vision-annotation-tool-a-universal-approach-to-data-annotation

https://c.onepanel.io/onepanel-demo/profile

Install

  • Nvidia-Docker
https://pythonawesome.com/a-web-based-tool-which-helps-to-annotate-video-and-images-for-computer/

Tutorials

  • YouTube - Nikita Manovich
https://www.youtube.com/user/nmanovic/videos
  • Annotation Adventures: A Concept Tutorial in Adding Labels to Image Datasets
https://www.linkedin.com/pulse/annotation-adventures-concept-tutorial-adding-labels-image-kacoroski/

OpenCV

  • Object Detection Using OpenCV
https://johnallen.github.io/opencv-object-detection-tutorial/


Data managment

  • big number of images (performance)
https://github.com/opencv/cvat/issues/151
  • Not compressing images
https://github.com/opencv/cvat/issues/204
  • Image storage in server
https://github.com/opencv/cvat/issues/95

Install

https://github.com/opencv/cvat/blob/develop/cvat/apps/documentation/installation.md

  • Install Docker
Update pip
sudo apt-get install python3-setuptools
sudo apt-get install -y python3-pip
sudo pip3 install --upgrade pip
sudo pip install --upgrade pip
Clone CVAT source code from the GitHub repository.
cd /home/regogr
If git not installed: sudo apt-get install -y git
git clone https://github.com/opencv/cvat
Edit yml file - set volumes
If not use default volumes
Edit docker-compose.yml file see Install Balder
Build docker images by default. It will take some time to download public docker image ubuntu
16.04 and install all necessary ubuntu packages to run CVAT server.
cd cvat
sudo docker-compose build


Run docker containers. It will take some time to download public docker images like postgres
10.3-alpine, redis:4.0.5-alpine and create containers.
sudo docker-compose up -d


You can register a user but by default it will not have rights even to view list of tasks. Thus you should create a superuser. A superuser can use an admin panel to assign correct groups to the user. Please use the command below
sudo docker exec -it cvat bash -ic 'python3 ~/manage.py createsuperuser'
Superuser: django
Email: rego.granlund@ri.se
Password django
Choose login and password for your admin account. For more information please read Django documentation.
Google Chrome is the only browser which is supported by CVAT. You need to install it as well. Type commands below in a terminal window
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
sudo apt-get update
sudo apt-get install -y google-chrome-stable
Open the installed Google Chrome browser and go to http://localhost:8080. Type your login/password for the superuser on the login page and press the Login button. Now you should be able to create a new annotation task. Please read the CVAT user's guide for more details.

Uninstall

Delete all images

Stop and remove all processes/containers

Stop process
sudo docker stop ContainerID
Delete process container
sudo docker rm ContainerID


Deleta all cvat images befor reinstall

Funkad
sudo docker rmi -f cvat
Funkade inte.
sudo docker rmi -f nginx
sudo docker rmi -f ubuntu
sudo docker rmi -f postgres
sudo docker rmi -f redis
Deletede med id nummer
sudo docker rmi -f

Deleta install folder

sudo rm -rfv /home/regogr/cvat

Delete data folders

sudo rm -rfv /var/lib/docker/volumes/cavt_cvat_data
sudo rm -rfv /var/lib/docker/volumes/cavt_cvat_db
sudo rm -rfv /var/lib/docker/volumes/cavt_cvat_keys
sudo rm -rfv /var/lib/docker/volumes/cavt_cvat_logs
sudo rm -rfv /var/lib/docker/volumes/cavt_cvat_models

No Docker

https://github.com/opencv/cvat/blob/master/CONTRIBUTING.md

Run

User guid [https://github.com/opencv/cvat/blob/develop/cvat/apps/documentation/user_guide.md CVAT

Start CVAT
In Google Chrome browser, go to http://localhost:8080.
Superuser: django
Password django
Data disk on 10.112.46.3
obdb on optimus.local fielddata/20190604152150/04/DCIM/
smbclient -L 10.112.46.3/obdb -U regogr
sudo mkdir /mnt/obdb
sudo mount -t cifs -o user=regogr //10.112.46.3/obdb /mnt/obdb
sudo mkdir /mnt/cvat
sudo mount -t cifs -o user=regogr //10.112.46.3/cvat /mnt/cvat

Admin

  • Django administration
http://localhost:8080/admin/
Add User
http://localhost:8080/admin/
Press ACCOUNTS
Press Email adresses - add
Press Magnifier (Förstoringsglaset)
Press Add user
Username: rego
Password:
Set User Rights
Did work after some attempts
Logout
Login
http://localhost:8080/admin/
In action list select user
name: Rego
name: Granlund
mail: rego.granlund@ri.se
Active: Yes
Staff: Yes
Superuser: Yes
Groups: Add all
Press Save

Volumes

The volumes containing the data are located at.

/var/lib/docker/volumes

sudo chmod 777 /var/lib/docker
sudo chmod -R 777 /var/lib/docker/volumes

Log

A log is saved at.

Example
/var/lib/docker/volumes/cvat_cvat_data/_data/2 cd /var/lib/docker/volumes/cvat_cvat_data/_data

Docker files

/home/rego/cvat

???

/home/rego/cvat/docker-compose.yml

Create New Task

https://github.com/opencv/cvat/blob/develop/cvat/apps/documentation/user_guide.md#creating-an-annotation-task

Fields

  • Name:
  • Labels:
Example
plants @select=type:undefined,Charlock,CreepingThistle,WildTurnip,MugWort,Dandelion,UnknowWeed,BlackBindweed,CurledDock,ScentlessMayweed,Fat-hen,Cleavers,CommonFumitory,Rape,Colt's-foot,HairyTare ~radio=quality:good,bad ~checkbox=dead:false
  • Bug Tracker:
  • Data set Repositury:
  • Use LFS:
If the annotation file is large, you can create a repository with LFS support.
LFS is a feature of git (https://git-lfs.github.com/).
If you enable it CVAT will store your annotations as LFS files in provided git repository.
Otherwise it will store them as usual text files.
  • Source:
  • Z Order:
  • Over Lap size:
  • Segment size:
  • Image Quality:
  • Start Frame:
  • Stop Frame:
  • Frame Filter:
  • Select Files:

Local Test

In Google Chrome browser, go to http://localhost:8080.

Use data: .../obdb/work/Amanda/Annotations/Klar 20_20190604152150_05 (2).xml ...

  • Name: RegoTest1
  • Labels:
plants @select=type:undefined,Charlock,CreepingThistle,WildTurnip,MugWort,Dandelion,UnknowWeed,BlackBindweed,CurledDock,ScentlessMayweed,Fat-hen,Cleavers,CommonFumitory,Rape,Colt's-foot,HairyTare ~radio=quality:good,bad ~checkbox=dead:false
  • Bug Tracker:
  • Data set Repositury:
  • Use LFS:
  • Source:
  • Z Order:
  • Over Lap size:
  • Segment size:
  • Image Quality:
  • Start Frame:
  • Stop Frame:
  • Frame Filter:
  • Select Files:Selected 10 files

Did generate: http://localhost:8080/?id=1

Cvat Server

Connect

ssh
ssh -l regogr -p 23122 10.112.46.3


Shared folders
cd /mnt
cd /mnt/cvat


Test install 1 - Baldr

Install Folders

Data root
/mnt/cvat/baldr/
Configuration copy
/mnt/cvat/baldr/concopy


Local Browser

sudo apt install chromium-browser


ssh -Y -l regogr -p 23122 10.112.46.3 chromium-browser

Good to have

sudo apt install nmap nmap localhost nmap 192.168.122.31 nmap 192.168.122.31 -p8000-9000

wget 192.168.122.31:8080 docker-compose down nano docker-compose.yml docker-compose up -d

ConCopy

docker-compose.yml
sudo cp /home/regogr/cvat/docker-compose.yml /mnt/cvat/baldr/concopy
sudo mv /mnt/cvat/baldr/concopy/docker-compose.yml /mnt/cvat/baldr/concopy/docker-compose-orginal.yml
cd /mnt/cvat/baldr/concopy
sudo pico /home/regogr/cvat/docker-compose.yml

Install

Clone CVAT source code from the GitHub repository.
Install git if needed:sudo apt-get install -y git
git clone https://github.com/opencv/cvat

Edit docker-compose.yml

Backup
sudo cp /home/regogr/cvat/docker-compose.yml /mnt/cvat/baldr/concopy
sudo mv /mnt/cvat/baldr/concopy/docker-compose.yml /mnt/cvat/baldr/concopy/docker-compose-orginal.yml
Edit
sudo pico /home/regogr/cvat/docker-compose.yml
sudo cp /home/regogr/cvat/docker-compose.yml /mnt/cvat/baldr/concopy


cvat_db:/mnt/cvat/baldr/postgresql/data

cvat_data:/mnt/cvat/baldr/django/data
cvat_keys:/mnt/cvat/baldr/django/keys
cvat_logs:/mnt/cvat/baldr/django/logs
cvat_models:/mnt/cvat/baldr/django/models


Fredrik funkar 

#
# Copyright (C) 2018 Intel Corporation
#
# SPDX-License-Identifier: MIT
#
version: "2.3"

services:
  cvat_db:
    container_name: cvat_db
    image: postgres:10.3-alpine
    networks:
      default:
        aliases:
          - db
    restart: always
    environment:
      POSTGRES_USER: root
      POSTGRES_DB: cvat
    volumes:
      - cvat_db:/var/lib/postgresql/data

  cvat_redis:
    container_name: cvat_redis
    image: redis:4.0.5-alpine
    networks:
      default:
        aliases:
          - redis
    restart: always

  cvat:
    container_name: cvat
    image: cvat
    restart: always
    depends_on:
      - cvat_redis
      - cvat_db
    ports:
      - "8080:8080"
    build:
      context: .
      args:
        http_proxy:
        https_proxy:
        no_proxy:
        socks_proxy:
        TF_ANNOTATION: "no"
        AUTO_SEGMENTATION: "no"
        USER: "django"
        DJANGO_CONFIGURATION: "production"
        WITH_TESTS: "no"
        TZ: "Etc/UTC"
        OPENVINO_TOOLKIT: "no"
    environment:
      DJANGO_MODWSGI_EXTRA_ARGS: ""
      UI_PORT: 7080
      ALLOWED_HOSTS: "*"


    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
      - cvat_models:/home/django/models

  cvat_ui:
    container_name: cvat_ui
    image: nginx
    restart: always
    build:
      context: .
      args:
        http_proxy:
        https_proxy:
        no_proxy:
        socks_proxy:
      dockerfile: Dockerfile.ui
    environment:
      REACT_APP_API_PROTOCOL: http
      REACT_APP_API_HOST: localhost
      REACT_APP_API_PORT: 8080
    networks:
      default:
        aliases:
          - ui
    depends_on:
      - cvat
    ports:
      - "7080:80"

volumes:
  cvat_db:
  cvat_data:
  cvat_keys:
  cvat_logs:
  cvat_models:

http://10.112.46.3:23188/admin/ 


#
# Copyright (C) 2018 Intel Corporation
#
# SPDX-License-Identifier: MIT
#
version: "2.3"

services:
  cvat_db:
    container_name: cvat_db
    image: postgres:10.3-alpine
    networks:
      default:
        aliases:
          - db
    restart: always
    environment:
      POSTGRES_USER: root
      POSTGRES_DB: cvat
    volumes:
      - cvat_db:/mnt/cvat/baldr/postgresql/data

  cvat_redis:
    container_name: cvat_redis
    image: redis:4.0.5-alpine
    networks:
      default:
        aliases:
          - redis
    restart: always

  cvat:
    container_name: cvat
    image: cvat
    restart: always
    depends_on:
      - cvat_redis
      - cvat_db
    ports:
      - "8080:8080"
    build:
      context: .
      args:
        http_proxy:
        https_proxy:
        no_proxy:
        socks_proxy:
        TF_ANNOTATION: "no"
        AUTO_SEGMENTATION: "no"
        USER: "django"
        DJANGO_CONFIGURATION: "production"
        WITH_TESTS: "no"
        TZ: "Etc/UTC"
        OPENVINO_TOOLKIT: "no"
    environment:
      DJANGO_MODWSGI_EXTRA_ARGS: ""
      UI_PORT: 7080

    volumes:
      - cvat_data:/mnt/cvat/baldr/django/data
      - cvat_keys:/mnt/cvat/baldr/django/keys
      - cvat_logs:/mnt/cvat/baldr/django/logs
      - cvat_models:/mnt/cvat/baldr/django/models

  cvat_ui:
    container_name: cvat_ui
    image: nginx
    restart: always
    build:
      context: .
      args:
        http_proxy:
        https_proxy:
        no_proxy:
        socks_proxy:
      dockerfile: Dockerfile.ui
    environment:
      REACT_APP_API_PROTOCOL: http
      REACT_APP_API_HOST: localhost
      REACT_APP_API_PORT: 8080
    networks:
      default:
        aliases:
          - ui
    depends_on:
      - cvat
    ports:
      - "7080:80"

volumes:
  cvat_db:
  cvat_data:
  cvat_keys:
  cvat_logs:
  cvat_models:

Run

http://10.112.46.3:23180/


Error

Error when running sudo docker-compose build
Command step
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
Processing triggers for dbus (1.10.6-1ubuntu3.4) ...


Error
gpg: keyring `/tmp/tmpv4i455ob/secring.gpg' created
gpg: keyring `/tmp/tmpv4i455ob/pubring.gpg' created
gpg: requesting key ED8E640A from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpv4i455ob/trustdb.gpg: trustdb created
gpg: key ED8E640A: public key "Launchpad PPA for Doug McMahon" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK
gpg: keyring `/tmp/tmpy0p4jqzy/secring.gpg' created
gpg: keyring `/tmp/tmpy0p4jqzy/pubring.gpg' created
gpg: requesting key ED8E640A from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpy0p4jqzy/trustdb.gpg: trustdb created
gpg: key ED8E640A: public key "Launchpad PPA for Doug McMahon" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK
Error
You are using pip version 8.1.1, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Current default time zone: 'Etc/UTC'
Local time is now:      Tue Nov 26 12:00:58 UTC 2019.
Universal Time is now:  Tue Nov 26 12:00:58 UTC 2019.
Error
Step 13/21 : RUN npm install
 ---> Running in ca75760cb59b
npm WARN deprecated fsevents@1.2.9: One of your dependencies needs to upgrade to fsevents v2: 1) Proper nodejs v10+ support 2) No more fetching binaries from AWS, smaller package size
npm WARN deprecated left-pad@1.3.0: use String.prototype.padStart()
npm WARN deprecated circular-json@0.3.3: CircularJSON is in maintenance only, flatted is its successor.

> core-js@3.4.2 postinstall /tmp/cvat-core/node_modules/core-js
> node -e "try{require('./postinstall')}catch(e){}"

Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!

The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
> https://opencollective.com/core-js 
> https://www.patreon.com/zloirock 

Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)

npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN acorn-jsx@5.1.0 requires a peer of acorn@^6.0.0 || ^7.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN cvat-core.js@0.1.0 No repository field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.9 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
Error
Step 14/21 : WORKDIR /tmp/cvat-ui/
 ---> Running in b42babfe132c
Removing intermediate container b42babfe132c
 ---> 4d181675b73f
Step 15/21 : RUN npm install
 ---> Running in bbb23d04b751

> core-js@2.6.10 postinstall /tmp/cvat-ui/node_modules/core-js
> node postinstall || echo "ignore"

Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!

The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
> https://opencollective.com/core-js 
> https://www.patreon.com/zloirock 

Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)


> nodemon@1.19.4 postinstall /tmp/cvat-ui/node_modules/nodemon
> node bin/postinstall || exit 0

Love nodemon? You can now support the project via the open collective:
 > https://opencollective.com/nodemon/donate

npm WARN cvat-ui@0.1.0 No repository field.
npm WARN cvat-ui@0.1.0 scripts['server'] should probably be scripts['start'].
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.9 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

Not Working

  • No home directory, To create home folder I did run: sudo mkhomedir_helper regogr
  • Command history does not work when connecting with ssh
  • scp does not work, did try running on my computer, not from 10.112.46.3
sudo scp -p 23122 regogr@10.112.46.3:/home/regogr/cvat/docker-compose.yml /home/rego
sudo scp -p 23122 regogr@10.112.46.3:docker-compose.yml /home/rego
  • sshfs does not work
psftp regogr@10.112.46.3:23122
  • tab completion does not work


Connect to //10.112.46.3/obdb /cvat
Funkar
sudo mkdir /mnt/obdb
sudo mount -t cifs -o user=regogr //10.112.46.3/obdb /mnt/obdb
Funkar inte
sudo mkdir /mnt/cvat
sudo mount -t cifs -o user=regogr //10.112.46.3/cvat /mnt/cvat

Work Files

  • smb://optimus.local/obdb/fielddata/20190604152150/05/DCIM/100GOPRO/iframes
  • smb://optimus.local/obdb/work/Amanda/Annotations
Retrieved from "http://www.c3learninglabs.com/mw/index.php?title=Dev:AI/CVAT&oldid=10807"