Shell-Script to publish data to MQTT

MrAnderson — Wed, 10 Nov 2021 20:54:47 GMT

Hello,

I created a small script to publish the data of all endpoints to a MQTT server .

Inspired by @SorinB and the nice small PHP script and the help of @CustomerCareMichael to open my eyes 😉 I ended up in this setup.

What you need:

any kind of Linux server (I'm using a RaspberryPi)
json_pp, jq, curl, sed, grep, cat, sort, tr are the commands I'm using in the script
mosquitto_pub is not necessary when publishing the data to something different (used at the end of the script)

What you may need:

mosquitto_pub
MQTT server (Mosquitto prefered)
Telegraf to feed a timeseries database
InfluxDB as a timeseries database
Grafana to visualize the data

Prerequisites:

Create a developer account and an API-Key at https://developer.viessmann.com/de/clients
Create a challenge id with this small tool: https://tonyxu-io.github.io/pkce-generator/
User 1. and 2. and your user and password used in the ViCare app and put them into the small script in lines 8 (client id), 10 (challenge id), 11 (user) and 12 (password)
Line 13 can be modified to have a prefix like "viessmann-testing" to get started and when you are ready change it to "viessmann". This prefix is used for the topic to be published to MQTT
Line 14 is a directory where all the working files are stored, this can be /dev/shm or /tmp, it will be created when the scripts starts.
Line 15 to 18 are the credentials and host:port parameter for MQTT server
Line 19 is the number of the device queried in (getDeviceId). May be you have more than one device like I have but 1 is a good start. Check the file devices.json in the directory provided in line 14

Now you can run the script on the command line and when it works you can add it to the crontab and execute every 5 minutes. It is also possible to use Telegraf to transport the values into an InfluxDB and Grafana to query that database 🙂

#!/bin/bash ############################# # Constants to change ############################# export client_id="client_id" export redirect_uri="http://localhost:4200/oauth-callback" export challenge_id="challenge_id" export email="username from ViCare App" export password="password" export prefix="prefix if necessary" export dirPrefix="/dev/shm/$prefix" export mqttHost="127.0.0.1" export mqttPort="1883" export mqttUser="mqtt" export mqttPassword="mqtt" export number=1 mkdir -p $dirPrefix >/dev/null 2>&1 ############################################################################ # Show a banner ############################################################################ banner() { echo "+------------------------------------------+" printf "| %-40s |\n" "`date`" echo "| |" printf "|`tput bold` %-40s `tput sgr0`|\n" "$@" echo "+------------------------------------------+" } ############################################################################ # Get access token or create new one when last login older than 1400 minutes # otherwise refresh token when token older than 59 minutes ############################################################################ getAccessToken() { export auth_url="https://iam.viessmann.com/idp/v2/authorize" export scope="IoT%20User%20offline_access" export response_type="code" export token_url="https://iam.viessmann.com/idp/v2/token" export grant_type="authorization_code" export countLoginHtml=$(find $dirPrefix -cmin -59 -name login.html | wc -l) if [ $countLoginHtml -eq 0 ]; then # file older 1400 minutes, get new access token curl -c $dirPrefix/cookies.txt -H "Content-Type: application/x-www-form-urlencoded" \ -X POST "$auth_url?client_id=$client_id&redirect_uri=$redirect_uri&response_type=$response_type&code_challenge=$challenge_id&scope=$scope" \ -u "$email:$password" \ -o $dirPrefix/login.html >/dev/null 2>&1 export code=$(grep $redirect_uri $dirPrefix/login.html | sed -E 's/(^.*code=)([^"]*)(.*)/\2/g') curl -b $dirPrefix/cookies.txt --basic -c $dirPrefix/cookies.txt -H "Content-Type: application/x-www-form-urlencoded" \ -X POST "$token_url" \ --data-urlencode "grant_type=$grant_type" \ --data-urlencode "code_verifier=$challenge_id" \ --data-urlencode "client_id=$client_id" \ --data-urlencode "redirect_uri=$redirect_uri" \ --data-urlencode "code=$code" \ -o $dirPrefix/access_token.json >/dev/null 2>&1 fi export countAccessToken=$(find $dirPrefix -cmin -59 -name access_token.json | wc -l) if [ $countAccessToken -eq 0 ]; then # file older 59 minutes, get new access token with refresh token export refresh_token=$(sed -E 's/^.*"refresh_token":"([^"]).*/\1/g' $dirPrefix/access_token.json) curl -b $dirPrefix/cookies.txt --basic -c $dirPrefix/cookies.txt -H "Content-Type: application/x-www-form-urlencoded" \ -X POST "$token_url" \ --data-urlencode "grant_type=refresh_token" \ --data-urlencode "client_id=$client_id" \ --data-urlencode "refresh_token=$refresh_token" \ -o $dirPrefix/access_token.json >/dev/null 2>&1 fi export access_token=$(sed -E 's/^\{"access_token":"([^"]*).*/\1/g' $dirPrefix/access_token.json) echo $access_token } ############################################################################ # Just show the user profile in pretty print json format ############################################################################ showUserProfile() { banner "Show user profile" export profile_url="https://api.viessmann.com/users/v1/users/me?sections=identity" curl -b $dirPrefix/cookies.txt --basic -c $dirPrefix/cookies.txt -H "Authorization: Bearer $(getAccessToken)" \ -X GET "$profile_url" \ -o $dirPrefix/userprofile.json >/dev/null 2>&1 cat $dirPrefix/userprofile.json | json_pp } ############################################################################ # Get the installation id ############################################################################ getInstallationId() { if [ ! -f $dirPrefix/installation.json ]; then export installation_url="https://api.viessmann.com/iot/v1/equipment/installations" curl -b $dirPrefix/cookies.txt --basic -c $dirPrefix/cookies.txt -H "Authorization: Bearer $(getAccessToken)" \ -X GET "$installation_url" \ -o $dirPrefix/installation.json >/dev/null 2>&1 fi export installationId=$(sed -E 's/^.*"id":([^,]*).*/\1/g' $dirPrefix/installation.json) echo $installationId } ############################################################################ # Get the gateway serial ############################################################################ getGatewaySerial() { if [ ! -f $dirPrefix/gateway.json ]; then export gateway_url="https://api.viessmann.com/iot/v1/equipment/gateways" curl -b $dirPrefix/cookies.txt --basic -c $dirPrefix/cookies.txt -H "Authorization: Bearer $(getAccessToken)" \ -X GET "$gateway_url" \ -o $dirPrefix/gateway.json >/dev/null 2>&1 fi export gatewaySerial=$(sed -E 's/^.*"serial":"([^"]*).*/\1/g' $dirPrefix/gateway.json) echo $gatewaySerial } ############################################################################ # Get the device id ############################################################################ getDeviceId() { if [ ! -f $dirPrefix/devices.json ]; then curl -b $dirPrefix/cookies.txt --basic -c $dirPrefix/cookies.txt -H "Authorization: Bearer $(getAccessToken)" \ -X GET "https://api.viessmann.com/iot/v1/equipment/installations/$(getInstallationId)/gateways/$(getGatewaySerial)/devices" \ -o $dirPrefix/devices.json >/dev/null 2>&1 fi export deviceNumber=$1 export deviceId=$(cat $dirPrefix/devices.json | json_pp | grep '"id"' | sed -n "${deviceNumber}p" | sed -E 's/^.*"id" : "([^"]*).*/\1/g') echo $deviceId } ############################################################################ # Get features ############################################################################ getFeatures() { export deviceNumber=$1 curl -b $dirPrefix/cookies.txt --basic -c $dirPrefix/cookies.txt -H "Authorization: Bearer $(getAccessToken)" \ -X GET "https://api.viessmann.com/iot/v1/equipment/installations/$(getInstallationId)/gateways/$(getGatewaySerial)/devices/$(getDeviceId $deviceNumber)/features" \ -o $dirPrefix/features-$deviceNumber.json >/dev/null 2>&1 } getFeatures $number ############################################################################ # publish the data to MQTT ############################################################################ cat $dirPrefix/features-$number.json | jq '.data[].feature' | tr -d '"' | sort -u > $dirPrefix/feature.dat cat $dirPrefix/feature.dat | while read feature do cat $dirPrefix/features-$number.json | jq '.data[] | select(.feature == "'$feature'") | .properties | keys[]' | tr -d '"' | sort -u > $dirPrefix/topic.dat cat $dirPrefix/topic.dat | while read topic do export TOPIC=$(echo $feature.$topic | sed 's/\./\//g') cat $dirPrefix/features-$number.json | jq '.data[] | select(.feature == "'$feature'") | .properties.'$topic'.value' > $dirPrefix/$feature.$topic.dat #curl -d @$dirPrefix/$feature.$topic.dat $mqttUrl/$TOPIC mosquitto_pub -h $mqttHost -p $mqttPort -u $mqttUser -P $mqttPassword -t $prefix/$TOPIC -f $dirPrefix/$feature.$topic.dat done done

You can see an example how it may looks like. I'm not finished yet, still some endpoints missing in the Grafana and the API seems to be incomplete. I couldn't find all the data the ViCare is showing..

// Andreas

Betreff: Shell-Script to publish data to MQTT

FloE — Tue, 05 Jul 2022 10:18:30 GMT

Hallo Andreas,

ich habe versucht, dein Skript auf meinem Raspberry Pi zu nutzen und bin gemäß deiner wirklich tollen und ausführlichen Anleitung vorgegangen.

Das Problem ist, dass offenbar die Authentifizierung nicht klappt. Im vom Skript erzeugten File access_token.json steht lediglich "{"error:"internal server error"}".

Im File devices.json steht unter anderem "statusCode":401,"errorType":"UNAUTHORIZED","message":"Token provided in request is expired or invalid."

Ich habe die "manuelle" Authentifizierung nach der Viessmann-Doku hinbekommen und konnte über den RPi auch schon manuell curl-Request senden und habe entsprechende Response erhalten. Demnach sollten die Daten für clientID, challengeID, User und Passwort korrekt sein.

Hast du hier vielleicht noch einen Tip für mich?

Danke und viele Grüße

Flo

Betreff: Shell-Script to publish data to MQTT

MrAnderson — Tue, 05 Jul 2022 10:23:30 GMT

Moin Flo,

Ich vermute eher das User/Passwort ein paar lustige Zeichen enthält die vom Script fehlinterpretiert werden.

Typische problematische Zeichen wären " ! ( $ &

Grüße

Andreas

Betreff: Shell-Script to publish data to MQTT

FloE — Tue, 05 Jul 2022 10:40:15 GMT

Hi Andreas,

du bist mein persönlicher Held des Tages! 😀

Es war tatsächlich eines der Zeichen im Passwort. Passwort geändert und nun funktioniert es tadellos.

Tausend Dank dir.

Viele Grüße

Flo

Betreff: Shell-Script to publish data to MQTT

petkow — Fri, 19 May 2023 15:37:00 GMT

Hallo Andreas,

ist das Script noch aktuell ?

In der Datei access_token.json steht {"error":"invalid-token-request"}

und es gibt im Passwort kein Sonderzeichen.

Ich versuche mit der API meine VX3 Solaranlage auszulesen.

Thema "Betreff: Shell-Script to publish data to MQTT" in Getting started programming with the Viessmann API

Shell-Script to publish data to MQTT

Betreff: Shell-Script to publish data to MQTT

Betreff: Shell-Script to publish data to MQTT

Betreff: Shell-Script to publish data to MQTT

Betreff: Shell-Script to publish data to MQTT