Enphase Envoy-S “Data Scraping”.

If you are interested in other Enphase information the following other pages may also be of interest:
Reverse Engineering the Enphase Installer Toolkit
Enphase Envoy-S Open Ports!

I’ve recently had to interface with an EnPhase Envoy Solar PV system. The annoying thing is the lack of documentation.

The API for the “cloud service” exists and is well documented but the API for the local device itself doesn’t seem to exist.

There is one tiny one page document that seems to suggest you can get some data from the device. The information gleaned from their example API ” http://device.ip.address.here/api/v1/production ” is poor.

{
 "wattHoursToday": 6641,
 "wattHoursSevenDays": 6641,
 "wattHoursLifetime": 6669,
 "wattsNow": 90
}

First.. it gives far LESS information than you can just retrieve from the main web page for the device on the LAN. Secondly the wattsNow field doesn’t seem to match the information displayed on the main page of the device. The wattsHoursSevenDays is also slightly off. Third.. it doesn’t even update regularly. You can’t have a system that detects dips in production due to cloud cover etc.

However. I did find some other sensible sources for information – and information that updates at an even more regular rate than the devices main website!

When you visit the site on the device running firmware D4.2.27 it makes the following requests:

http://10.0.0.177/backbone/application.js?version=04.02.43
This URL contains a lot of references to other json files that I need to research. See further down in this post for the other URLs exposed.

http://10.0.0.177/home.json
This json file contains the Database size / utilisation, date and time on the device, connection status and settings and update status.

http://10.0.0.177/production.json
Contains most useful information.. Production and (if fitted) consumption data!

{
   "production":[
      {
         "type":"inverters",
         "wNow":74,
         "whLifetime":6815.014722222222,
         "readingTime":1470250044,
         "activeCount":14
      },
      {
         "type":"eim",
         "activeCount":1,
         "whLifetime":6704.458,
         "whLastSevenDays":6676.458,
         "whToday":6676.458,
         "wNow":66.236,
         "rmsCurrent":1.179,
         "rmsVoltage":246.433,
         "reactPwr":276.774,
         "apprntPwr":290.494,
         "pwrFactor":0.23,
         "readingTime":1470250044
      }
   ],
   "consumption":[
      {
         "type":"eim",
         "activeCount":1,
         "whLifetime":8025.821,
         "whLastSevenDays":7899.821,
         "whToday":7899.821,
         "wNow":2641.386,
         "varhLeadToday":2822.701,
         "varhLagToday":1810.03,
         "vahToday":9018.68,
         "varhLeadLifetime":2951.701,
         "varhLagLifetime":1922.03,
         "vahLifetime":9473.68,
         "rmsCurrent":11.924,
         "rmsVoltage":246.458,
         "reactPwr":-284.009,
         "apprntPwr":2938.817,
         "pwrFactor":0.9,
         "readingTime":1470250044
      }
   ]
}

http://10.0.0.177/inventory.json
Also contains some very useful information. Detailed status about the state of each micro-inverter. So far I’ve seen the following states:

– “envoy.global.ok”
– “envoy.cond_flags.pcu_ctrl.commandedreset”
– “envoy.cond_flags.pcu_ctrl.dc-pwr-low”
– “envoy.cond_flags.obs_strs.discovering”
– “envoy.cond_flags.obs_strs.failure”
The full conversion table on what these status messages mean in english is at the end of this article.

The full output of the page is as follows:

[
   {
      "type":"PCU",
      "devices":[
         {
            "part_num":"800-00356-r04",
            "installed":"1470228526",
            "serial_num":"REDACTED",
            "device_status":[
               "envoy.cond_flags.pcu_ctrl.commandedreset",
               "envoy.cond_flags.pcu_ctrl.dc-pwr-low"
            ],
            "last_rpt_date":"1470249596",
            "admin_state":1,
            "created_date":"1470228526",
            "img_load_date":"1424997903",
            "img_pnum_running":"520-00045-r01-v01.22.00",
            "ptpn":"540-00087-r01-v01.22.00",
            "producing":true,
            "communicating":true,
            "chaneid":1627392273,
            "device_control":[
               {
                  "gficlearset":false
               }
            ]
         },
         {
            "part_num":"800-00356-r04",
            "installed":"1470228529",
            "serial_num":"REDACTED",
            "device_status":[
               "envoy.cond_flags.obs_strs.discovering"
            ],
            "last_rpt_date":"1470249602",
            "admin_state":1,
            "created_date":"1470228529",
            "img_load_date":"1424997903",
            "img_pnum_running":"520-00045-r01-v01.22.00",
            "ptpn":"540-00087-r01-v01.22.00",
            "producing":false,
            "communicating":true,
            "chaneid":1627392529,
            "device_control":[
               {
                  "gficlearset":false
               }
            ]
         },
         {
            "part_num":"800-00356-r04",
            "installed":"1470228533",
            "serial_num":"REDACTED",
            "device_status":[
               "envoy.global.ok"
            ],
            "last_rpt_date":"1470249613",
            "admin_state":1,
            "created_date":"1470228533",
            "img_load_date":"1424997903",
            "img_pnum_running":"520-00045-r01-v01.22.00",
            "ptpn":"540-00087-r01-v01.22.00",
            "producing":true,
            "communicating":true,
            "chaneid":1627392785,
            "device_control":[
               {
                  "gficlearset":false
               }
            ]
         }
      ]
   }
]

All of this means you can make your own system that (in my case) refreshes generation and consumption status up-to-the-second!

A couple of notes: The Consumption metering / data seems to give strange low or even minus values if you fetch the json file more than roughly once per second. (Possibly because it works on a counter and the math goes wrong?)
During nighttime the Production metering goes into a minus value, for example -7 Watts. This is in fact the correct reading. The Envoy-S javascript filters out minus values and ensures a 0 is displayed. Cheeky!
The panel count on the main production.json file is the count of associated panels and not the currently generating panels. I’ve used the inventory.json file to count the number of panels “producing”, I refresh this information every 3 minutes instead of every second.

solar status

solar panel status

My next steps are to investigate the JavaScript and see what other pages are accessible. I also need to see if I can reverse engineer, find out or ask the installer for the “Installer Login” password.

I also wish there was public documentation. No google searches for the status strings or other data I’ve found in the JSON files have returned any results to do with Enphase / SolarPV.

The backbone/application.js file exposes the following sub-files on the devices web server.
– /admin/lib/admin_dcc_display.json
– /ivp/tpm/capability
– /datatab/event_dt.rb?start=0&length=153
– /info.xml
– /ivp/meters
– /installer/pcu_comm_check
– /api/v1/production/inverters
– /prov
– /ivp/peb/newscan
– /installer/profiles/index.json
– /installer/profiles/details.json
– /installer/profiles/inverters_status.json
– /installer/profiles/set_profile.json
– /admin/lib/admin_pmu_display.json
– /ivp/peb/reportsettings
– /ivp/tpm/select
– /ivp/tpm/tpmstatus
– /ivp/tpm/parameters
– /admin/lib/dba.json
– /admin/lib/security_display.json
– /admin/lib/date_time_display.json?tzlist=1&locale=en
– /admin/lib/network_display.json
– /api/v1/production/inverters
– /admin/lib/wireless_display.json?site_info=0

Full status messages system strings to plain english:

        cond_flags: {
            acb_ctrl: {
                bmuhardwareerror: "BMU Hardware Error",
                bmuimageerror: "BMU Image Error",
                bmumaxcurrentwarning: "BMU Max Current Warning",
                bmusenseerror: "BMU Sense Error",
                cellmaxtemperror: "Cell Max Temperature Error",
                cellmaxtempwarning: "Cell Max Temperature Warning",
                cellmaxvoltageerror: "Cell Max Voltage Error",
                cellmaxvoltagewarning: "Cell Max Voltage Warning",
                cellmintemperror: "Cell Min Temperature Error",
                cellmintempwarning: "Cell Min Temperature Warning",
                cellminvoltageerror: "Cell Min Voltage Error",
                cellminvoltagewarning: "Cell Min Voltage Warning",
                cibcanerror: "CIB CAN Error",
                cibimageerror: "CIB Image Error",
                cibspierror: "CIB SPI Error"
            },
            obs_strs: {
                discovering: "Discovering",
                failure: "Failure to report",
                flasherror: "Flash Error",
                notmonitored: "Not Monitored",
                ok: "Normal",
                plmerror: "PLM Error",
                secmodeenterfailure: "Secure mode enter failure",
                secmodeexitfailure: "Secure mode exit failure",
                sleeping: "Sleeping"
            },
            pcu_chan: {
                acMonitorError: "AC Monitor Error",
                acfrequencyhigh: "AC Frequency High",
                acfrequencylow: "AC Frequency Low",
                acfrequencyoor: "AC Frequency Out Of Range",
                acvoltage_avg_hi: "AC Voltage Average High",
                acvoltagehigh: "AC Voltage High",
                acvoltagelow: "AC Voltage Low",
                acvoltageoor: "AC Voltage Out Of Range",
                acvoltageoosp1: "AC Voltage Out Of Range - Phase 1",
                acvoltageoosp2: "AC Voltage Out Of Range - Phase 2",
                acvoltageoosp3: "AC Voltage Out Of Range - Phase 3",
                agfpowerlimiting: "AGF Power Limiting",
                dcresistancelow: "DC Resistance Low",
                dcresistancelowpoweroff: "DC Resistance Low - Power Off",
                dcvoltagetoohigh: "DC Voltage Too High",
                dcvoltagetoolow: "DC Voltage Too Low",
                dfdt: "AC Frequency Changing too Fast",
                gfitripped: "GFI Tripped",
                gridgone: "Grid Gone",
                gridinstability: "Grid Instability",
                gridoffsethi: "Grid Offset Hi",
                gridoffsetlow: "Grid Offset Low",
                hardwareError: "Hardware Error",
                hardwareWarning: "Hardware Warning",
                highskiprate: "High Skip Rate",
                invalidinterval: "Invalid Interval",
                pwrgenoffbycmd: "Power generation off by command",
                skippedcycles: "Skipped Cycles",
                vreferror: "Voltage Ref Error"
            },
            pcu_ctrl: {
                alertactive: "Alert Active",
                altpwrgenmode: "Alternate Power Generation Mode",
                altvfsettings: "Alternate Voltage and Frequency Settings",
                badflashimage: "Bad Flash Image",
                bricked: "No Grid Profile",
                commandedreset: "Commanded Reset",
                criticaltemperature: "Critical Temperature",
                "dc-pwr-low": "DC Power Too Low",
                iuplinkproblem: "IUP Link Problem",
                manutestmode: "In Manu Test Mode",
                nsync: "Grid Perturbation Unsynchronized",
                overtemperature: "Over Temperature",
                poweronreset: "Power On Reset",
                pwrgenoffbycmd: "Power generation off by command",
                runningonac: "Running on AC",
                tpmtest: "Transient Grid Profile",
                unexpectedreset: "Unexpected Reset",
                watchdogreset: "Watchdog Reset"
            },
            rgm_chan: {
                check_meter: "Meter Error",
                power_quality: "Poor Power Quality"
            }
        }

Some more info about errors can be found on page 45 of the manual.
CommandedReset Recommended Action: No action is required; it will automatically resume normal operation momentarily. Description: The microinverter was reset, either following a successful software download or by user command.

DcPowerTooLow Recommended Action: This condition should correct itself. No action is required. Description: This condition may occur at sunrise or sunset, while the modules are covered with snow, or during extreme weather. This event indicates that sunlight levels are too low for effective production. Once sunlight levels increase, the microinverter resumes power production and this event message will clear.

DcVoltageTooLow Recommended Action: This is usually a normal condition during hours of low light and at dawn and dusk. No action is required. Description: The microinverter reports that DC input voltage from the PV module is too low. If this condition does not clear during hours of full day

PowerOnReset Description: The microinverter has powered on after having DC disconnected.

Advertisements
This entry was posted in Uncategorized. Bookmark the permalink.

81 Responses to Enphase Envoy-S “Data Scraping”.

  1. Pingback: Enphase Envoy-S Open Ports! | thecomputerperson

  2. RRM says:

    Bought the envoy-s 3 weeks ago and having the same difficulties in reading the local data.
    Found several scripts for running man-in-the-middle option or scripts for OpenHAB, but haven’t found the information you wrote.
    Unfortunately I have not that much information in the production.json, as you demonstrate.
    I bookmark this link and follow your progress.

  3. Thank you for commenting. What firmware version is your Envoy-s running ?
    Does it show that you have micro-inverters connected and “producing” on the envoy-s web interface?

  4. RRM says:

    @license Backbone-Envoy 4.2.43 (2016-05-20)

  5. RRM says:

    Although the Enphase site clearly demonstrates the exact number of producing inverters, the only output I get from production.json is:
    {“production”:[{“type”:”inverters”,”wNow”:46,”whLifetime”:477193.87611111114,”readingTime”:1470855222,”activeCount”:20}]}

  6. Is that all you see in the production.json? In my case there is a “type”:”eim”, section and also a “consumption” section.
    You should also see if /inventory.json contains anything interesting too.

    I’ve not yet managed to work out how to get individual inverter generation statistics.

    This evening my Envoy-S seems to have crashed or hung! This will be interesting to try and work out how to solve. It is responding to pings but the web interface is not returning any data.

    Update: In the end I just had to turn off the power to the Envoy-S and turn it back on again.

    Do you have an Envoy or an Envoy-S? The one I have is running firmware “D4.2.27 (b13066)” with a “Software Build Date” of “Jun 03, 2016 1:17 AM” which is strange. Your comment seems to show you have a _newer_ firmware but with a build date after the build date of my older firmware which doesn’t make much sense.
    Also the request to the javascript file that my firmware makes references 04.02.43.. even more weird.

  7. RRM says:

    It’s an Envoy-S (EU version). The inventory page contains output which is more or less identical. Unfortunately the production.json not.

  8. Not sure what to suggest :(

  9. Paul says:

    Thanks for the head start.

    I have > D4.4.x

    when prompted for username:password are envoy:nnnnnn where nnnnnn=last 6 digits of serial number

    This works for wifi and diagnostics from home but not other screens :(

    try
    /home?locale=en&classic=1
    /home?locale=en&classic=2

    /production.json?details=1
    /production.json?details=2 etc (might be interval records?)

    to do

    The installer toolkit can see all inverter details without a password – but how? Time to sniff some traffic.

  10. Thanks for the info. I’ve yet to try the installer toolkit and prod the buttons on the Envoy. Will it work while the Envoy is on the normal network? (My one is hard wired / ethernet).

  11. Paul says:

    Yes and shows live inverter level data. It can connect to the envoy wirelrss ap but also works if device is on the same subnet. The app queries the presence of the envoy and it responds with its serial number which appears in the app. Touch this and your in.

    You can see this arp traffic using wireshark but not http from app ??? May be limitation of windows on promiscuous mode can see http access from browser np..

  12. I will give it a go and post an update. It will be at least a few weeks before I get a chance though. Too many things to do and not enough time.

  13. Paul says:

    also try

    /info
    /ivp/meters

    Authentication of installer app process

    https://s3.amazonaws.com:443/mobile-versions.enphaseenergy.com/app_versions.json
    /info
    app uses api key (based on info? eg S/N) and pings. This was different to the web API key!
    xxapi/xamarin.com/api/ping?apikey=XXXXXXXXXXXXXXXXXXXXXXXX
    there is some traffic back and forth inc some binary
    x-pong
    /api/collected?apikey= XXXXXXXXXXX&currentTime=xxxxxxx

    /prov returns unauthorised
    /prov again gets a response. It contains authorisation below

    GET /prov HTTP/1.1
    Accept: application/json
    Authorization: Digest username=”installer”, realm=”enphaseenergy.com”, nonce=”xxxxxxxxxxxxxxxx”, uri=”http://192.168.0.x/prov”, response=”xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”, qop=auth, nc=00000001, cnonce=”xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”
    Host: 192.168.0.x
    Connection: Keep-Alive
    Accept-Encoding: gzip
    User-Agent: okhttp/2.7.

    hmmmmm……. digest access auth

  14. The information you are seeking about the energy each inverter is generating appears to be returned at this URL:

    /api/v1/production/inverters

    It requires authentication. In my case I was able to replay an authentication request that I had sniffed 13 hours earlier! So if you can sniff a successful authentication header from the installer toolkit request _to that specific page_ you may just be able to replay the authentication line from the header for eternity!

    [{
            "serialNumber": "REDACTED1",
            "lastReportDate": 1472215582,
            "lastReportWatts": 221,
            "maxReportWatts": 244
        }, {
            "serialNumber": "REDACTED2",
            "lastReportDate": 1472215588,
            "lastReportWatts": 226,
            "maxReportWatts ": 248
        }]

    While playing about I did manage to get /home.json to spit out:

        "alerts": [{
            "msg_key": "lwui.home.warnings.cross_domain_traffic"
        }],

    Interesting.

  15. RRM says:

    Unfortunately the amazon link of Paul isn’t working at the moment.
    I also tried to get access through the installer toolkit, but with the lack of passwords, it wasn’t successful.

    /api/v1/production/inverters requires name and password. I get exact the same data of my 20 inverters by filling in envoy and the last 6 numbers of the serial number.

  16. I’ve fixed paul’s link – it was missing the https on the http:// bit.
    Installer toolkit just let me right on on my system without the need to authenticate. Yet if I visit the same URLs on the browser it requires authentication.
    As Paul noted, the app generates the authentication details somehow.
    If you can sniff the details it seems the final authentication http header can be reused for eternity. It seems the envoy doesn’t check for expiry of the nonce.

    Sadly whatever details are generated are specific to your unit so I can’t share my http request with you and have you authenticate against your Envoy with it :(

  17. Paul says:

    Ypu shouldnt need user .. pass with the installer toolkit.

    It finds the envoy on the local lan or you can use the envoy ap lan.

    The s/n will appear when its found. Just tap on this.

  18. The Installer Toolkit application is a Xamarin (Mono) .net cross platform app. This means, when extracted, you get a load of (as far as I’m aware) windows compatible .dll files.
    The authentication is likely handled by “assemblies\Enphase.InstallerToolkit.dll”.

    Next step – I’ve got to learn how to reverse engineer .net dll files :< This might be beyond my abilities.

  19. Pingback: Reverse Engineering the Enphase Installer Toolkit | thecomputerperson

  20. RRM says:

    Succeeded in getting access through the installer kit.
    Tried to sniff the procedure with wireshark, which produced some files with http 1.1 or application/json information in it.
    Tried to search for keys/passwords, but my lack of knowledge of wireshark makes it a slow process. I’ll keep on trying with your suggestions.

  21. Dave says:

    Anyone know how to pass the username/password to the envoy-s api via http post? I’m looking to grab the panel level info, and I can do so via:
    http://192.168.1.10/api/v1/production/inverters
    (192.168.1.10 is the wifi IP of my envoy-s)

    But it needs a username of ‘envoy’ and password of the last 6 digits of the serial number, which it pops up and requests. I’ve tried:
    http://192.168.1.10/api/v1/production/inverters&user=envoy&password=123456
    (where 123456 is the last 6 digits of my envoy-s)
    which returns:
    {
    “status”: 404,
    “error”: “”,
    “info”: “Resource /api/v1/production/inverters&user=envoy&password=123456 not found”,
    “moreInfo”: “”
    }

    I’ve also tried the same thing but instead of & I used ?, and it popped up with the password prompt as if I wasn’t using them.

    Anyone figured this one out yet?

  22. What code are you using to scrape the data? Normally you just pass it extra parameters such as..

    xmlhttp.URL(“http://192.168.1.10/api/v1/production/inverters”)
    xmlhttp.Credentials(“envoy”,”034222″)
    xmlhttp.Go()

    (Very rough example, not working code!).

  23. Dave says:

    I was using a post–just pasting into the web page.
    I believe that using a format similar to an ftp site works. For instance:
    http://envoy:123456@192.168.1.10/api/v1/production/inverters

    I need to test it on a separate computer.

    Once I have that working, I’ll figure out what language I want to learn to do this…right now I know perl, but I think I should probably learn python.

    Thanks for the reply…good to know there is a way to do it, and that I’m not barking up a dead tree!

  24. I’d approach it the other way around. Mash out the code and get it to authenticate and dump the resulting page.

    I’ve not got any experience in Perl but as far as I can see LWP takes authentication options:

     use LWP 5.64;
      my $browser = LWP::UserAgent->new;
      $browser->credentials(
        '10.0.0.177:80',
        'realmnamehere',
        'usernamehere' => 'passwordhere'
      );
    my $response = $browser->get($url);

    Perl looks pretty horrible from the code I’ve just seen, I expect Python is a far better bet :)

  25. wharwood says:

    I’ve been working to programmatically authenticate the inverter production page via Matlab. The built in authentication options don’t work and the method suggested above of using http://user:pass@xxx doesn’t work either.

  26. Do you have an Envoy or Envoy-S? My Envoy-S doesn’t need authentication for /inventory.json or /production.json so I can’t test against my own system and help you with code :(

  27. wharwood says:

    I have an Envoy-S. I too don’t need authentication for /production but the real data I want is in /production/inverters. That is password protected.

  28. Ah yes, that one is password protected but I’ve not yet figured out how to calculate the password from the envoy serial number. The older Envoy firmware was “installer” and the last 6 digits of the serial number (as far as I remember) but the newer firmware or Envoy-S seems to have a dynamically generated password.
    For more information on my password generation probing see:
    https://thecomputerperson.wordpress.com/2016/08/28/reverse-engineering-the-enphase-installer-toolkit/

  29. wharwood says:

    I must have an older unit then because user: envoy pass: XXXXXX (last 6 of serial) works for me.

  30. Doesn’t work for me :( I am on Software Version D4.2.27 (b13066)

  31. Dave says:

    I was able to get to this page by using the ‘envoy’ username and last 6 digits of the serial number. Try:
    http://envoy:123456@192.168.1.10/api/v1/production/inverters
    (where you substitute the last 6 of your serial and your IP number)

    The only odd thing is that it pops up with a prompt that you have to click yes to, not sure how that’ll work out programatically.

    I haven’t had time to look into this, since I posted last.

  32. That one works for me with the envoy/XXXXXX details!

    {
        "serialNumber": "REMOVED",
        "lastReportDate": 1475688141,
        "lastReportWatts": 0,
        "maxReportWatts": 259
      },
  33. wharwood says:

    Matlab returns 401 on Dave’s link for me. Glad you that login info worked for you. I’m running software version R4.2.33 (b68db1)

  34. I can download Matlab and test here. I’ve never used it before – what are the lines of code you have that make the request ?

  35. wharwood says:

    It all revolves around the built in function “webread”. Documentation is here: https://www.mathworks.com/help/matlab/ref/webread.html Truthfully, Matlab is probably not the best language for this. It’s just the one they taught me in school. It’s also expensive.

  36. Dave says:

    My system has build D4.2.27 (b13066), for reference

  37. wharwood says:

    It interesting that we have what appears to be very different firmware builds. I have the Envoy-S Metered (ENV-S-AM1-120). I would guess you both do not have the metered version.

  38. I have a consumption and production clamp meter that came with the Envoy-S and the data read out shows on the Envoy page.

  39. It looks like matlab is just sending “Basic” authentication in a single request and not triggering a Digest authentication (First a non-authenticated request needs to be made in which the envoy provides a salt / nonce which matlab [or other software] then hashes the authentication details with to produce a unique authentication string before making the second authenticated request with this string).

    The following might help but I can’t actually see any code in that example about authentication even though it claims it tackles Digest auth.
    https://uk.mathworks.com/help/matlab/matlab_external/send-http-message.html

    Wireshark is a great tool – Install it and use the filter “host 10.0.0.177” where 10.0.0.177 is the IP of your envoy. Make a request / run your matlab script and you can then right click on a request and go to Follow –> TCP Stream to see if matlab sent a Basic or Digest authentication line.

  40. wharwood says:

    Just had a breakthrough. First, I took my own advice and got away from Matlab. I grabbed Visual Studio and started writing a console app in C#. The following code grabs the json file:

    var envoy = new WebClient { Credentials = new NetworkCredential(user, pw) };
    var response = envoy.DownloadString(url);

  41. jam says:

    It was only a matter of time before someone blogged about this stuff! I currently have OpenHAB polling my envoy-s production.json and logging production and consumption data, and also activating heating etc based on available power! :) The APIs they provide are USELESS….!

    I’ve also had my envoy-s metered crash twice now since I began polling it every few seconds or so. It required a hard reset. I have no evidence of course that it wouldn’t have crashed anyway, but it is annoying. Has anyone else noticed this, or has anyone had their envoy-s crash when not polling it? (you get an email from enlighten saying it is not reporting)

  42. I’ve certainly had it crash twice. Both times I thought it was due to my polling and it seemed to crash in an extended amount of time when I changed the polling to 2 seconds instead of 1 second etc.. but I don’t recall changing my polling code since and I’ve not had it crash (yet).
    If you poll the Envoy consumption data more than once per second you get totally screwed up values (minus values and the 7 day consumption total starts counting down!).

  43. Having said that – it crashed again today. Lots of other random services still respond but port 80 doesn’t ever load and the Envoy stops reporting to the cloud service. Boring. Wish I could reboot it remotely :<

  44. Jam: I’ve been testing. I thought maybe the crash was triggered by either.
    1) Requests being sent too quickly to the Envoy
    2) A hard-limit on the number of requests that the envoy can handle before it runs out of ram, temp filesystem space or something
    3) Race condition

    The number of requests I seemed to send before it crashed was around 604,800 however.. I’ve been soak testing it with a script that has been hammering the Envoy with over 100 requests per minute and I’ve reached well over 800,000 requests and it still hasn’t crashed.
    There goes my theory.

    Which files are you fetching every second? I was testing with /production.json
    I wish I had SSH or some other debug access to the device to either restart it remotely or, even better, figure out what is causing the lock up.

  45. jam says:

    Hi tcp, I really appreciate you hammering your system in the name of science!

    I am only fetching /production.json, once every 10 seconds on a little wifi display I built to have in the kitchen, and once every 30s on OpenHAB.

    I really think getting the installer password used by the installer app could possibly open up some options for getting further access to the box.

    I’m surprised they don’t have some kind of watchdog running on it to restart the service – it does stop it uploading to enlighten, and does stop logging data when it “crashes” so it’s kind of important….

  46. emdeex says:

    Some good info here, thanks. I’m using Splunk to fetch and display the data from my Envoy-S … there’s a short forum post here: https://answers.splunk.com/answers/390296/how-to-get-splunk-to-fetch-json-data-from-enphase.html#comment-451860

  47. Over the last week my Envoy updated itself from “Software Version D4.2.27 (b13066)” to “Software Version D4.4.89 (5e7c1d)”.
    I now find that the web interface (html and everything) takes many minutes to load up and the json files take a similar amount of time. Sometimes it takes 1.4 minutes, sometimes it takes 3.5 minutes, sometimes it takes 5 minutes to just return one of the json pages.

    The web interface now mentions batteries which I believe have recently been launched as an Enphase product in several countries.

    Anyone else had this firmware? Anyone else having page load delay problems?

  48. jam says:

    Same firmware has upgraded itself here – no delay problem. Also haven’t had any crashes for ages!

  49. Thanks.. I will monitor and see what is going on. Seems to be ok for a bit and then goes weird, possibly when I request inventory.json and production.json in very quick succession.

  50. Right. Not sure what has changed in the firmware / envoy back end but I’ve figured out what is going on. My code is quite aggressive: It checks for /production.json every second _but_ has a 1 second no response timeout. This is normally ok but..

    Also, every 3 minutes, I check for /inventory.json. It seems that sometimes /inventory.json is taking more than 1 second to respond with data (sometimes takes 600ms, sometimes takes up to 6 seconds) and is “blocking” so other requests get queued / delayed.

    My code then, at the next second, requests /production.json but as /inventory.json is still processing – it times out, then requests it again a second later.
    It seems that prior requests get queued still even though the TCP connection has been ended [RST, ACK (forcefully ended)] with the timeout. This ends up running away and the envoy appears to have locked up. If you stop _all_ requests and leave it for about 10-20 minutes, it recovers.

    You can reproduce this problem by loading up one of the .json files in your browser and constantly spamming the Refresh -> Stop -> Refresh -> Stop button (without waiting for the page to load each time). After about 20 or so goes at that you should see that your final request and ones shortly after take quite a long time to be responded to by the envoy.

    Something has changed with the way the envoy handles requests or how quickly it can respond to requests.. compounded by my aggressive code.
    I guess I now need to code, my side, failure detection and make my code “back off” so that the envoy can recover or fulfil all pending requests (double the delay until the next request, then double it again if it fails a second, third, fourth (etc.) time).

    I hope that the new firmware, with this quirk, does at least fix the total lockups that required a power cycle to recover from :D

  51. No crash or lockup since the firmware update!
    Yay. I no longer have to power cycle the envoy weekly.

  52. Ken Clifton says:

    Thanks for your work on this! I wish that I had an Envoy-S. My two older Envoys don’t have many of the items mentioned. In case you are interested, I created a Python script to archive the event log on the older Envoy version — it may work with your Envoy-S… Short link: https://goo.gl/kWHcBJ

  53. Cam says:

    Hi,
    I can get a fair bit of data following you instructions including panel production, but what program did you use to make your own system and refresh the data?
    Thanks

  54. I’m using asp classic and built in logic to cache the response for X seconds before making another request.
    This means I can have multiple displays or systems (automatic space heater that uses up spare energy instead of exporting) polling my ASP page multiple times per second! but my ASP only “bothers” the Envoy-S every 3 seconds and returns that cached result until it expires.
    See my previous comments (above) about how the Envoy really doesn’t cope well with lots of requests in quick succession.

    ASP Classic was my choice because I knew I could develop something rapidly. Anything else would also work fine, PHP, Perl, Python or even a simple bash script using wget and grep.

  55. Cam says:

    Thanks for the quick response. Any chance you would be able to share the script/programming you wrote for your own system which I could then adapt. I’m not great with writing my own. Also I would probably only request every 10 seconds so I don’t upset my envoy.
    Thanks again.

  56. Difficult to share without knowing what it is going to be used for. My code spits out javascript which probably won’t be suitable for whatever you want to use it for. And my code is ASP classic which requires an IIS / Windows server.

  57. Cam says:

    No worries. I’ll work on it from another angle.
    Thanks.

  58. Dave says:

    Ken’s python script does not work with the current envoy-s firmware. It gets both an unauthorized error, and it looks like the envoy login doesn’t allow access to the event log, at least at that location, even with the envoy:last6ofsn login.

  59. Ken Clifton says:

    Thanks for checking that Dave. You can try downloading the event log directly to see if the location is correct — it is at — http://%5Benvoy ip address]//datatab/event_dt.rb

    I wish that I had an Envoy-S to test with. Does anyone know if the Envoy-S will work with the older M190, M215 and M250 series inverters?
    The fact that it exposes the production information makes it very attractive to me…

  60. Compatible Micro-inverters.. a convenient table is here:
    https://enphase.com/en-uk/products-and-services/envoy/family
    Looks like the M215 and M250, yes.. M190 isn’t even mentioned.

    On the Envoy-S you need to issue the event log request with start= and length= variables for it to return information. For example /datatab/event_dt.rb?start=0&length=153

    I’m still stuck on the reverse engineering of the password hashing function :( Shame.

    Off-topic: I can find at least 688 Envoys just “open on the internet”.. I do hope they’ve been designed securely! Otherwise another potential IoT disaster.

  61. Ken Clifton says:

    Yes, the whole IoT thing is a disaster waiting to happen. My two Envoy(s) are behind multiple firewalls.
    Since I don’t have the “S” model, I can’t do any testing. My Python script is fully commented so you can see where the request to the event_dt.rb has a whole bunch of parameters being passed. I do understand about the start and length on your “S” version.
    I wish you luck with the password hashing. It is a shame they don’t expose the data with a password known to the user.

  62. Ken Clifton says:

    BTW, to get the parameters I did the to get the network monitor in Firefox, then copied the exact request made by the Eventlog page on the Envoy. I noted in the script the parameters that I have changed with effect — some it seems to ignore.

  63. Ken Clifton says:

    The keys were stripped out: control and shift and q

  64. Matthew says:

    Guys if I want to pass my user id envoy and password – last six digits of the Envoy-s to read the individual inverter / panel data – does anyone know the format of how I need to pass these parameters? My code has issues passing these credentials.

    If I key the local device URL into Chrome 192.168.0.12/api/v1/production/inverters and enter envoy and 012345 when it prompts for user id and password it accepts this and shows all the relevant details:

    {
    “serialNumber”: “121440006251”,
    “lastReportDate”: 1484971459,
    “devType”: 1,
    “lastReportWatts”: 172,
    “maxReportWatts”: 255
    },
    {
    “serialNumber”: “121440009849”,
    “lastReportDate”: 1484971423,
    “devType”: 1,
    “lastReportWatts”: 220,
    “maxReportWatts”: 251
    },

    etc

    My html file is executing the following JSON code

    $.getJSON(“http://envoy:012345@192.168.0.12/api/v1/production/inverters”, function(data) { …}

    BTW – is the code for the screen on the first page shareable? Love to see how youo did what you have!

    Many thanks,

    Matthew

  65. It looks like getJSON might be jquery? If so it seems that jQuery can’t do Digest authentication and will likely just be sending Basic authentication headers.
    http://stackoverflow.com/questions/5288150/is-digest-authentication-possible-with-jquery

    This looks like an addon for jquery that might help:
    https://www.openhub.net/p/digestj
    (I’m no expert in jquery so the above might be wrong or out of date).

    Ref – the code from the first screen shot.. it is in ASP Classic which pretty much nobody uses (or should be using) so probably not worth sharing.

  66. Dave says:

    http://trilema.com/2017/the-incidental-humiliation-of-obamas-clean-energy-policies-marc-andreessens-internet-of-farts-and-other-such-comedic-gold-brick

    This guy hangs out with people who attempt to factor RSA keys. It appears that early Envoys all use the same SSH RSA hostkey from Debian – and it isn’t prime!

  67. Interesting article you link to – doesn’t seem to be the specific one about the keys used but does have some interesting stuff in it.
    Seems that they are looking at a tsv file of some energy generated (doesn’t say source or what other info the file contains) but have deduced that there isn’t much energy being produced vs. the energy to manufacture solar generation kit.

    A bit amazed by their findings / calculation. The stuff on my roof is certainly more than covering my daily energy usage and I’m exporting more than I use. I’m not sure on the energy cost to produce the equipment in the first place but I can’t believe the guys claim that they are not economical. “To recap : “solar power” costs more in energy alone to merely produce the hardware (we’re not counting the minerals, just the energy to assemble all the stuff together) than it will ever output over its lifetime.”
    “For every Watt of energy an installed solar panel ever produces, it will have consumed about three Watts of energy. You are actually saving two Watts by not putting one in. ”

    I really don’t believe that but I’ve yet to investigate their calculations in depth.

  68. Dave says:

    Yeah if they are correct with their calculations then whoever installed that solar array did a poor job. My array is covering my energy use as well, although admittedly it will still take quite awhile to pay off. I’m using my excess electricity to mine cryptocurrencies which helps.

    The more interesting finding to me is the link to the SSH RSA key. I figured out how to extract and convert the key on my Enovy-S and it is not the same as what these guys claim to have found. So it appears this is a problem with old Envoys, not new ones.

  69. Jason says:

    Hi jam… you said earlier:

    I currently have OpenHAB polling my envoy-s production.json and logging production and consumption data, and also activating heating etc based on available power! :)

    Can you expand on this please? Their doco is confusing me — I don’t see anywhere where it can take input data and graph (or do something) with it?

  70. jam says:

    That’s right, you won’t find anything specific in the docs about it. But all the information is actually there – you want to poll your envoy, not use the enphase binding.

    So I just use the http binding to poll the envoy-s and parse the json returned. You parse out the solar production and consumption values into an openhab “Number” variable. Then just switch on the rrd4j “persistence” for those Number variables.

    Then generate a chart from the rrd4j. Or action what you will based on the current power produced using the rules engine.

    http://docs.openhab.org/addons/bindings/http1/readme.html

  71. Jason says:

    Ta for the info. Yeah, I don’t trust the Enphase cloud API… I have suspicions that they’re munging data.

    I’m putting together some Python, using SQLAlchemy to pull all data from the Envoy, storing it locally… and will probably end up graphing it using plot.ly and Flask as the frontend.

    I’m stoked as well, because production/inverters is working for me with creds as suggested above ;) so that’ll be interesting to see which panels do what, at which times of day.

  72. Ken Clifton says:

    Thanks for all the good work here! I upgraded one of my Envoy(s) to the “s” model. That has allowed me to develop some Python for it. I created a repository for the Inventory script on GitHub today. A post with that link and a screenshot can be seen at: http://www.kenclifton.com/wordpress/2017/06/envoy-s-local-solar-inverter-inventory-and-status/

    As well as running on standard Python version 3, it also works in the Pythonista App on iOS.

  73. Matthew says:

    I have found the following code gets a lot further when I execute $.getJSON(“http://192.168.0.12/api/v1/production/inverters?locale=en&user=envoy&password=012345”, function(data) {

    Now that it seems to pass authentication I am encountering network error

    SCRIPT7002: XMLHttpRequest: Network Error 0x2efd, Could not complete the operation due to error 00002efd.

    I found this on both firefox and edge – on Chrome I recieve

    ‘Access-Control-Allow-Origin’ header is present on the requested resource. Origin ‘http://127.0.0.1:8000’ is therefore not allowed access. The response had HTTP status code 401.

  74. Access-Control-Allow-Origin is going to scupper you there. The envoy must be replying with that in the HTTP headers which then tells the browser to not allow inclusion of that data into your json / javascript.
    The only way around it is to get the envoy not to provide that header.

    If I were having this problem I’d build a simple php or aspx script to fetch the /inverters data and then spit it out again without the Access-Control-Allow-Origin header!.. or maybe make a scheduled task or crontab to wget the data into a file and then host that file on your own web server.

  75. Matthew says:

    Well I made a lot more progress by changing @.getJSON to $.getScript – it invokes the request user name and password and when supplied it does return success – and I can see the data returned to it in debug mode, the only issue I have is I log all the variable to console.log and somehow the data variable itself comes back as undefined. This may be becuase $.getScript is a ayschronous call – and by the time I log the headers, call success etc the data itself hasn’t yet been returned. So it seems a less challenging problem to tackle. In debug mode I can see a Get call is issued – all the headers and Params look correct and the Call stack shows 31 individual inverters data.

    My logic is:

    https://code.jquery.com/jquery-3.1.1.min.js

    table td.box {
    width: 250px;
    border-left: 1px solid #000000;
    border-right: 1px solid #000000;
    }
    table td.box div {
    background-color: #0000FF;
    }

    Solar Test

    $(function() {
    var url = “http://192.168.0.12/api/v1/production/inverters”;

    $.getScript ( url, function( data, textStatus, jqxhr ) {

    console.log( textStatus );
    console.log( jqxhr.status );
    console.log( data );
    console.log(“Load was performed”);

    var table = $(“Serial NumberLast Report DateStatusLastMax”).addClass(“table table-striped”);
    var totalLastReportWatts = 0;
    var totalMaxReportWatts = 0;

    $.each( data, function(index, item) {
    console.log( index );
    totalLastReportWatts += item.lastReportWatts;
    totalMaxReportWatts += item.maxReportWatts;

    var percent = ((item.lastReportWatts / item.maxReportWatts) * 100);
    var row = $(“”).html(“” + item.serialNumber + “” + new Date(item.lastReportDate) + ” ” + item.lastReportWatts + “” + item.maxReportWatts+ “”);
    table.append(row);
    });

    var totalsRow = $(“”).html(“” + totalLastReportWatts + “” + totalMaxReportWatts + “”);
    table.append(totalsRow);
    $(“#results”).append(table);
    });
    });

    Any idea how to get the data to be actually linked to the data field so I can parse it correctly?

  76. Matthew says:

    I also tried re-structuring my code to execute the GetScript first – which toggles the Enphase Envoy-s unit to ask for authentication via a pop up window in the browser running the html file. Once these details are passed the call progresses – but with the returned data being classed as undefined and failing the string parsing.

    But I then added a second GetJSON call immediately after the GetScript call to see if the Authentication details created for the Get Script call where inherited by the subsequent GetJSON call – unfortunately they are not and a CORS error is recorded, buts its a 401 error – failed authentication.

    In firefox Debugger – it appears the GetScript call is a JS call type – which fails with a 401 – but the credentials when entered into the pop up Window that appears then immediately causes the Envoy to return the correct data with a 200 status code.

    When the GetJSON call is launched it is a JS XHR request and it doesn’t trigger the credentials logic the GetScript call does, so it doesn’t get the required authentication token / credentials and it simply fails with a 401 status.

    * * * * * *

    I wonder if there is any way to have the GetJSON call trigger the Envoy to ask / launch the credentials pop up window the GetScript did – so it too could pass authentication?

  77. Matthew says:

    Hey Jam, if you have got it all working could you please share your code somewhere!

  78. jam says:

    Hey there,

    Well, I don’t like reinventing the wheel so as I said I used OpenHAB, and the http binding included.

    So in the openhab.cfg I set the http binding timeout to 3000ms:
    ############################### HTTP Binding ##########################################
    #
    # timeout in milliseconds for the http requests (optional, defaults to 5000)
    http:timeout=3000

    Then in the OpenHAB “Items” configuration file I make 3 items, one to hold the entire json from the envoy+s, and then one for production and one for consumption:

    String Enphase_JSON “JSON data from Enphase Envoy-S” {http=”<[http://192.168.0.4/production.json:30000:REGEX((.*))]"}
    Number Power_Consumption (Solar2)
    Number Solar_Production (Solar2)

    Then I make a rule in the OpenHAB “rules” config file to parse the JSON when it is updated in the Enphase_JSON String item, writing the results to the Power_Consumption and Solar_Production items:

    rule “Init”
    when
    Item Enphase_JSON changed
    then
    var String data = Enphase_JSON.state.toString
    var Number production_value = transform(“JSONPATH”,”$.production[?(@.type==eim)][0].wNow”, data)
    var Number consumption_value = transform(“JSONPATH”,”$.consumption[?(@.type==eim)][0].wNow”, data)
    postUpdate(Solar_Production, production_value as String)
    postUpdate(Power_Consumption, consumption_value as String)
    end

    and _that’s it_. You can then turn on persistence for the two Items to store the production and consumption values over time, graph them, write other rules to switch other openHAB items on and off based on the values, like power switches, etc etc etc… It just works.

    The downside is you have to learn OpenHAB, which might be worth it anyway IMHO! :)

  79. Matthew says:

    Thanks Jam – something new to learn! So are you parsing per panel / micro power – and if so did you use OpenHAB – to pass the Authenctication paramaters envoy + last 6 digits of the envoy to in your case 192.168.0.4/api/v1/production/inverters

    Many thanks – given me something new to research!

  80. jam says:

    Nah, I didn’t bother with the per panel data. I didn’t need it for my automation needs, although it would be interesting to graph my split array separately (9 panels NE, 9 panels NW). In reality though it actually doesn’t vary that much!!

    I’m anyway lucky enough to have access to enlighten manager to scratch the per panel itch, and besides if you’re too OCD, per panel data can drive you nuts. 1W difference here, 2W there.. it doesn’t really matter to be honest.

  81. I did python script which scrapes the envoy.local APi’s and combines from different json responses one json (the data I wanted) I use this to store to my db and show it on my magic mirror. Codes are free to use in case it helps somebody: https://github.com/teemuhirsikangas/magicaespeculo/blob/master/scripts/send_envoy.py

Comment on this topic

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s