Category: Programming

Picking the right language for the task..

DarshanaLeave a comment

In the past year I had to juggle multiple programming languages much more intensively. While it helped me to improve my skills and refresh, I also came to some conclusions.

Knowing to select a language for a task, knowing these languages and their capabilities is useful in RnD area as time and effort can be saved in writing the code and focus can be put to the actual research.

Below views are my personal opinions based on past experience. They might not be the most correct.

C/ C++

Best for high performance, real time work. Can be a hindrance for quick testing or prototyping. My usual setup is CMake + C++, commonly used for OpenCV projects, etc. I’m not very happy when it comes to things like REST calls, JSON objects…

IDE support can be sometimes bit annoying, esp. finding an open source one, they tend to hog memory or CPU. I use QT Creator, VS Code ( != visual studio**) and just plan text editors.

Yet when written well, its in it’s own class of beauty 😛

Python

After a long pause, I got back to using python. I paid more attention on code quality, learnt more of specific python3 matters.

Quite nice for quick prototyping, etc. Can handle many things well. Specially scientific calculations, matrices, etc got excellent support with libraries such as Numpy and Scipy. In essence, it is entirely possible to use Python in place of Matlab and can be quite faster and of course free of charge.

These libraries are good in performance too, thanks for nice support for calling C/ C++ API’s from python, computationally intensive algorithms can be written in C/C++ and wrap around Python. (C++ calling from Java was a nightmare in comparison). This same feature would be useful when providing Python API’s for C/C++ libraries or calling computationally intensive, optimized algorithms.

I’m not very convinced of production-level code, probably better to convert to Java or C++ (or use a wrapper?) when bundling. These questions mainly arise when dealing with proprietary code that has to be deployed to customers. There is bytecode based distributing option like Java, but I don’t know much about options.

Python definitely do better than JS for prototyping, esp. matrices, etc stuff.

JavaScript

Alright for quick prototyping with NodeJS. Being a born-in-browser language, it’s support for web and related things, async programming is nice.

However, for larger projects, consider writing in Typescript or similar typed language for better code quality and ease of debugging/ IDE support.

There is at least 2 libraries for everything, but finding a feature-complete, good ones isn’t that easy. Don’t even bother to do matrix calculations or “big-data”/ statistical stuff in this 😛

I rather stupidly implemented a camera calibration tool on JS since the debug framework was NodeJS + browser, pretty much wrote everything including a bug-freed Levenberg Marquardt solver.

Java

Despite “write once, run anywhere”, C++ is preferable on some ends (with many FOSS libraries available in most platforms). Java can do pretty much everything like C/ C++, but that doesn’t mean it should be used for everything!

However deploying in many platforms is easier with Java . Yet I don’t fully agree on “enterprise grade stuff is written in Java” story.

I believe Java usage in Enterprise environments got accelerated with Sun certifications, hardware, the very nature of JVM and there are many mature tools written in Java. My disagreement comes because there are better tools out there for certain jobs but some people cannot do anything without Java which can become a bottleneck in RnD environments.

Languages/ frameworks should be chosen for the task and situation, not by the fact the programmer or system architect is __ years experienced with Java based systems.

PHP

This used to be my de facto choice for server side web apps, rendering, etc. WordPress, Joomla, etc are written on PHP!

It’s quite fast (According to a recent benchmark I saw, PHp7.0 is only slower than -o2 C++ code. In the sense of variables typing, PHP is similar to Javascript and the code can look very messy.

Decent looking and readable code can be written in PHP with good use of classes and using well made frameworks. In terms of web-servers, Java tend to be generally slower in this area, and PHP is optimized and pretty much made for this job.

Conclusion

While it is possible to do pretty much everything from all these languages, it should be noted that all of these languages are written in C/ C++ in general – why? performance. But that doesn’t mean everything should be written in C/ C++.

Some people like to write everything in Java, C/C++ or Python, what I wanted to point out in this rather entangled and not-so verbose post is that tools are there for convenience and to get the job done!!

Don’t use a sledgehammer to crack a nut

File services with Dreamfactory (file creation)

DarshanaLeave a comment

This and the following posts will cover the areas about the file API that was not clearly covered in the Dreamfactory Documentation. The focus on this article is about file & folder creation.

Contents

  1. Create a file
    1. Set content via JSON Request Body
    2. Set content via Multi-part form upload (file upload)
    3. Download to server from URL
  2. Create a folder
  3. Combine folder and file creation

General Information

For all these requests, a JSON request body is used in the following basic format.

POST : Each element of resource array will define a File, Folder request.

  • type : file or folder
  • name : name of file
  • path : path to file
  • content_type : media type ie: text/json
  • content : contents of the file
{
    "resource": [
        {
            "name": "folder2",
            "path": "folder2",
            "type":"folder"
        },
        {
            "name": "folder2/file1.txt",
            "path": "folder2/file1.txt",
            "type":"file",
            "content_type": "text",
            "content":"gdfgdgdfgf"
        }
    ]
}

1. Create a file

1.1 One or multiple files with JSON data.

Type : POST

Note: Setting property “is_base64”: true will enable to upload images with “content” set as Base 64 encoded string.

Warning: Using Base64 encoding for large images is highly discouraged!!! (I tried uploading 18MP image and it was not worth the trouble). For these cases, go for direct file upload by multi-part form.

{
    "resource": [
        {
            "name": "folder2/file1.txt",
            "path": "folder2/file1.txt",
            "type":"file",
            "content_type": "text",
            "content":"gdfgdgdfgf"
        },
        {
            "name": "folder2/file2.txt",
            "path": "folder2/file2.txt",
            "type":"file",
            "content_type": "text",
            "content":"lsjfvgm"
        }
    ]
}

This will create two files in the folder names “folder2”. If the folder does not exist, an error will occur.

1.2 Multi-part form upload (file upload)

When I tried to upload some big images, base 64 encoding was not good for the performance at any level. Therefore direct file upload was the best option for my scenario. Unfortunately this was not clearly documented in the wiki.

Type : POST (multi part form)

I have tested and used the following three methods.

Method 1. Plain HTML + minimum Javascript Based.

This can be done in a few ways. Below example is from “test_rest.html” in dreamfactory to test REST calls. Of course the javascript can be completely dropped out if needed.

HTML

<form enctype="multipart/form-data" onsubmit="postForm(this)" action="/api/v2/system/user/" method="POST">
 <input type="hidden" name="app_name" value="admin" />
 <!-- MAX_FILE_SIZE must precede the file input field -->
 <input type="hidden" name="MAX_FILE_SIZE" value="3000000000000" />
 <!-- Name of input element determines name in $_FILES array -->
 Test importing users via file: <br/>
 <input name="files" type="file" />
 <br/>
 <br/>
 <input type="submit" value="Send File" />
 </form>

JS

function postForm(form){
    var jwt = $('#token').val(); //Session Token
    var apiKey= $('#app').val(); // API Key for app
    var url = $('#url').val(); //url for the REST call
    form.action = url+'?session_token='+jwt+"&api_key="+apiKey;
    // the token and api key can be sent as headers (if going for AJAX call)
}

Method 2. JQuery AJAX Based.

TODO

Method 3. Java (okHTTP) based

This method employs the already given Java SDK example for Dreamfactory. I have modified ImageUtils.java, ImageServices.java for my purpose. Original link https://github.com/dreamfactorysoftware/android-sdk

Note: This method was done to use on Java 8 on a PC. The method for Android is different and can be found in the api info.

ImageUtils.java

public void addImageFromLocalFile(String fileServiceName, String imageDir, String imageName, File imgFile, Callback<FileRecord> callBackFileRecord) {

 RequestBody requestBody = new MultipartBody.Builder()
 .setType(MultipartBody.FORM)
 .addFormDataPart("files", "imageName-1.png",
 RequestBody.create(MediaType.parse("image/png"), imgFile))
 .build();
 final ImageService imageService = DreamFactoryAPI.getInstance(App.SESSION_TOKEN).getService(ImageService.class);

 imageService.addLocalImage(fileServiceName, imageDir, imageName, requestBody).enqueue(callBackFileRecord);
 }

ImageService.java

@POST("{file_service_name}/{id}/{name}")
Call<FileRecord> addLocalImage(@Path(value = "file_service_name") 
 String fileServiceName, @Path(value = "id") 
 String contactId, @Path(value = "name") 
 String name, @Body RequestBody file);

1.3 Download to server from URL

This method is quite straightforward and explained in Dreamfactory Wiki;

http://wiki.dreamfactory.com/DreamFactory/Tutorials/Uploading_File#Example_-_Upload_a_JPEG_image_to_a_directory_called_images_using_storage_service_called_.27files.27

2. Create Folder

Creating a directory is similar to creating a file, the format is called FolderRequest. The difference is, you will be calling a directory in the API call. From the looks of it, this trick may work without specifying the exact folder, etc!!!

{
    "resource": [
        {
            "name": "folder2",
            "path": "folder2",
            "type":"folder"
        }
    ]
}

3. Combine Folder and File Creation

Same URL format in the above case, the only difference is you can ask Dreamfactory to create a folder and put files into it. First the file to be created must be specified, then the files to place inside the folder.

{
    "resource": [
        {
            "name": "folder2",
            "path": "folder2",
            "type":"folder"
        },
        {
            "name": "folder2/file1.txt",
            "path": "folder2/file1.txt",
            "type":"file",
            "content_type": "text",
            "content":"gdfgdgdfgf"
        }
    ]
}

Conclusion

The file API is quite nice and create a layer between the filesystem and our applications. You can easily switch to a cloud based storage or a different network drive without the users noticing it.

 

Dreamfactory – API Automation!

DarshanaLeave a comment

I came across Dreamfactory while I and a colleague of mine were searching for a REST API for PHP. In summary, this framework simplified a lot of setting up and development time! Specially this is an open source project and has enterprise support.

The most valuable feature I saw is ability to connect to almost all major database types and automatically generate the REST API calls. On top of that, this system offers role based authentication and a lot of features.

What you can do with Dreamfactory;

  • Connect to a database and get all necessary REST calls
  • User management, role based authentication, application level access control
  • Custom server side scripting with v8JS, PHP, Node.JS, Python
  • Auto generated API documentation with “try now” option (Based on swagger)

Setting up this framework need some practice and experience with the command line, however following the wiki articles will certainly do the job.

Performance : not so much! Depending on the server, the time to process a REST call may take up to a half a second or more.

You can try Dreamfactory on their trial accounts or you can clone the Git repo and set it up on a local machine or a hosted server.

Setting up GCC, CMake, Boost and Opencv on Windows

DarshanaLeave a comment

Background Story

For a project I’ve been working on, the need came to build the program to run on Windows OS. The project was written in c++ and used OpenCV and Boost libraries. For ease of configuration I employed CMake.

Despite the target being Windows, I was developing and testing everything under GNU/Linux 😛 , fortunately I managed to write the code with minimal amount of native unix API calls. For example, file handling was done via Boost Filesystem and so on.

Therefore the only consideration was running the CMake script in windows, using Visual Studio or GCC (via MinGW). First attempt was done with VS 2013, but compilation failed with a bug of VS c++ compiler related to c++ template classes and getting a later version was taking time. So I gave a try for gcc on windows!

Objectives

First I wanted to see if c++ building on windows with relative ease is possible. Next, I wanted to avoid the dependency of Visual Studio for the matter. This might be specially useful if you code for commercial work, but cannot afford to buy the license or simply dislike Visual Studio 😛

Step 1. Install MinGW

I went for MinGW-w64 (http://mingw-w64.org/doku.php) build since its more accepted and supports 64 bit. (Don’t expect citations justifying this 😀 ).

  1. Mingw Builds (http://mingw-w64.org/doku.php/download/mingw-builds) distribution was chosen as I didn’t want to install cygwin or win builds. Its a simple install. The following combination of settings worked for me.
    • Target Architecture – 64 bit (personal choice, depends on target system)
    • Threads – Win32 (Some recommend POSIX over Win32, However openCV build failed with mysterious problems with POSIX threads)
    • Exception – seh (I didn’t do much research here, just kept first available option
  2. Once the installation is complete, navigate to the install location and look for “bin” folder.
  3. Add that location to the PATH variable
    1. Control Panel -> System -> Advanced System settings -> Environment Variables -> system variables -> choose Path and click Edit
    2. Append the path to “bin” folder into the PATH variable. (There are tons of guides of how to do this)
  4. Open a command line (Start -> cmd.exe)
  5. Type the following commands it should show the version and other info 
    • gcc -v
    • This is merely used to confirm setting the PATH variable worked.
  6. Once gcc works, navigate to “bin” folder of MinGW install and make a copy of “mingw32-make” and rename it to “make”. (This executable provide “make”) This step is for convenience with CMake 😉

Step 2. Install CMake

  1. Download (https://cmake.org/download/)
  2. Run the installer
    • It’ll ask whether it is okay to include CMake binary location to the PATH variable – Tick yes, preferably system wide.
  3. Open a command line (cmd.exe) and run the following 
    • cmake --version
    • Provided everyting works, the output will show the CMake version.

Step 3. Build and Install OpenCV

  1. Download OpenCV from http://opencv.org/downloads.html
    • I built 3.1.0 with default options.
  2. Extract the archive (ie: D:\opencv-3_1_0)
  3. Open CMD and change directory to source folder of opencv. From this step, all commands would be executed through CMD unless otherwise noted. 
    • D:
    • cd D:\opencv-3_1_0\source
    • First command is not needed if OpenCV resides in C: for other partitions enter the partition name to change to that, then use the cd command. (I find this inconvenient 😛 )
  4. Run CMake. The main change is additional “MinGW Makefiles” parameter. Other than that, this step is pretty much going with standard OpenCV Documentation. Add necessary arguments fitting the needs!. 
    • cmake -G "MinGW Makefiles" [other arguments] ../build
    • “../build” at end pointed “build” directory as destination of MakeFile
  5. Once Cmake configures successfully, change to the build directory and execute make. 
    • cd ../build
    • make -j5
    • I used -j5 as my computer have 4 logical processors so 5 threads is well enough to fully load it! If the computer have 8 cores, use -j8 or -j9
    • Use the multithreaded compile option (-j5) with caution, some laptops tend to go into thermal shutdown with maximum load!!
  6. If the build complete successfully, next run make install 
    • make install
    • This step will finalize the install.

Step 3. Build and Install Boost

  1. Download and extract boost archieve from (http://www.boost.org/users/download/)
  2. Make sure to download the source package!
  3. Extract the archive, enter the directory from CMD.exe (example below) 
    • cd "D:\Program Files\boost\"
  4. Run the following commands 
    • bootstrap.bat gcc
    • b2 --build-dir=build cxxflags="-std=c++11" -j5 --with-filesystem --with-system define=BOOST_SYSTEM_NO_DEPRECATED toolset=gcc stage
    • Note the use of “toolset=gcc”, “-j5” options. These are self explanatory!
    •  “–with-<library_name>” flag  is used to explicitly include the necessary ones only. If you choose to build all libraries, then don’t specify this at all.
  5. If everything works fine, then Boost Build is complete!. Navigate to “stage” folder and go through the inner folders, there would be the compiled DLL files. (ie: libboost_system_xxx_mingw_xx.dll)

Step 4. Setting up CMake Script to work with windows

  1. Usage 
    • cmake -G "MinGW Makefiles" .
  2. I mashed up OpenCV and Boost CMake example scripts and some of my experiments to come up with the following CMake script.
  3. This script is a bare-bone CMake script, it has to be modified to suite your own project. I’m not an expert in CMake, so there would be room for improvement.
  4. I’ve tested the script with following configurations for the same exact project.
    • Kubuntu (16.04 LTS) with CMake 3.2.2, OpenCV 3.1.0, Boost 1.58
    • Windows 7 Professional, with CMake 3.6.0-rc_2, OpenCV 3.1.0, Boost 1.61
  5. The script is an updated version featured in my previous post on Boost, OpenCV, CUDA and CMake on Linux (https://tuxbotix.net/2016/03/18/nvidia-optimus-bumblebee-and-cuda-on-kbuntu-15-10/)
cmake_minimum_required(VERSION 3.2 FATAL_ERROR)

set(execFiles test.cpp)

if(WIN32)
 set(OpenCV_DIR "D:\opencv-3-10\build") #Change this
endif()

set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake/Modules/")

if(CMAKE_COMPILER_IS_GNUCXX)
 set (CMAKE_CXX_FLAGS "-std=c++11 -lm")
 if(WIN32)
 set(BOOST_ROOT "F:\Program Files\boost_1_61_0") #change this, critical!
 endif()
endif()

find_package( OpenCV REQUIRED )
FIND_PACKAGE( Boost 1.58 COMPONENTS filesystem system REQUIRED )
if(Boost_FOUND)
 include_directories(${Boost_INCLUDE_DIRS})
endif()
message(boost ${BOOST_INCLUDEDIR})

LIST(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}")
set(CMAKE_BUILD_TYPE Release)

add_executable (testApp ${execFiles})

target_link_libraries(testApp ${OpenCV_LIBS} ${Boost_LIBRARIES})

Possible Issues, Observations

I came across a few issues while setting up GNU Toolchain on Windows as well as configuring Boost, Opencv.

  1. CMake complains “CMake was unable to find a build program corresponding to “MinGW Makefiles”. CMAKE_MAKE_PROGRAM is not set.”
    • Cause : Seems to be CMake not recognizing “mingw32-make” as the make program despite CMake documentation saying it works! (https://cmake.org/cmake/help/v3.6/generator/MinGW%20Makefiles.html)
    • Workaround is making a copy of mingw32-make and rename it as “make”
    • The workaround may clash with existing “make” executable in the PATH. So take care!!
  2. When configuring Boost, “c1 is not recognized as an internal command”
    • Cause : Not specifying toolchain when executing bootstrap.bat and b2.exe
  3. Windows shows errors “libboost_system_xx_mingw_xx.dll” is not installed or “libopencv_imgproc310.dll is not installed” or similar errors
    • Cause : Windows cannot locate the DLL files.
    • Simplest fix is just copying the necessary DLL files and package them when distributing.
    • Warning : Always check legal matters (license agreement) before packing libraries that are not owned by you. Even if the libraries are open source, the license type may restrict distribution in binary format like this.

Nvidia Optimus, Bumblebee and CUDA on Kbuntu 15.10

Darshana1 Comment

I decided to write this post after experiencing a chain of weird events with setting up CUDA with ubuntu (Kubuntu,etc)!

I’ve used CUDA on OpenCV with Archlinux in 2014-2015 and it wasn’t too hard to get to work. But the story with Ubuntu is completely different 😛

First path is nvidia developer repo. That have own perils but you get latest CUDA version. (7.5). Second path is Ubuntu provided way which is much safer but not the latest (6.5).

Option 1 CUDA through nvidia repo. ( nvidia proprietary drivers + nvidia-primus)

Follow the guide at http://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/#axzz43DrWGyFP

They’ll replace ubuntu driver with their version (352.79 instead of 352.68 in my case)

Warning 1 :Do not reboot right now, otherwise you may come to a black screen and will have to boot from a live cd and chroot !! From my experience you have to run gpu-manager manually (explained below)

Warning 2: nvidia-prime will set nvidia chipset as the default** This doesn’t work very well and consume power (caused kde to crash with multiple monitors, strange font sizes, etc).

  • Use nvidia-settings and set intel as the primary chipset.
  • Login from the command line (alt + ctrl +f1);
  • Run the following commands, first will stop the DM (sddm for kubuntu, kde dropped KDM since kde 5) second will run gpu-manager that’ll go through the configuration.
  1. sudo systemctl stop sddm
  2. sudo gpu-manager
  • Observe the output of gpu-manager. Now you can reboot and see the results.
  • Make sure everything works fine (multiple monitors, etc).

Option 2 CUDA through Ubuntu Repo (nvidia proprietary + bumblebee or nvidia-primus)

Install nvidia drivers. Easiest path is using the “driver manager” software of ubuntu/ kubuntu. In kubuntu its accessible in System settings.

Note "Driver Management" icon in hardware section.
Note “Driver Management” icon in hardware section.
Choose the nvidia proprietary driver. (352 recommended)
Choose the nvidia proprietary driver. (352 recommended)

Next install the following packages: nvidia-cuda nvidia-cuda-toolkit

sudo apt-get install nvidia-cuda-dev nvidia-cuda-toolkit

Now run “nvcc -V” and see if the compiler runs.

Getting CUDA to work with cmake and gcc

I prefer to use cmake script for OpenCV projects so I’ll explain that method. Other options are easily found on internet.

If you observe closely, the compatibility matrix shown at nvidia website, maximum supported gcc version at this time is 4.9 with CUDA 7.5 Now the issue is Ubuntu 15.10 have gcc 5+.

So the first fix had to be install gcc 4.9 and point nvcc to gcc 4.9. In cMake scripts, the following declarations worked for me. In addition I had to specify some more info. Some people suggest editing nvcc.profile but I didn’t bother, I was already using cmake for the opencv projects!

set(CUDA_TOOLKIT_ROOT_DIR "/usr/local/cuda")
set(CUDA_HOST_COMPILER "/usr/bin/gcc-4.9")
set(CUDA_CUDART_LIBRARY "${CUDA_TOOLKIT_ROOT_DIR}/lib64/libcudart.so")
find_package(CUDA 7.5)

The first two lines found via opencv cmake config and the third had to be added after cmake complained “missing CUDA_CUDART_LIBRARY”.

Now everything should work fine;

If you get “invalid device ordinal” when running CUDA apps, the reason is the driver seems not to load properly on resuming from sleep. Dmesg will show this as “gpu falling off the bus”. Currently I couldn’t find a fix for the matter, I guess editing the config of nvidia-primus or bumblebee may help.

CUDA devicequery

Some history without dates 😛

Initially the only solution for graphics switching and keeping the nvidia chipset from overheating or acting strange required some hacking of ACPI calls. Then this project called Bumblebee emerged (http://bumblebee-project.org/). Thanks for the bumblebee daemon, it was possible to have proper power management. Some time later, they released a kernel module called “bbswitch”. This module make life even easier by automatically enabling power management.

Since the beginning I went with bumblebee + nvidia propriatery drivers, so as usual I went that way with ubuntu.

By now, Nvidia has released their version of optimus switching for Linux, the package is known as nvidia-prime. In past, the only alternative was bumblebee. I’m yet to see which works better, but I’m not much concerned as the only use of nvidia in linux is for CUDA based stuff. On windows however, it sees enough action ;).