Ignoring Negative Difference within Energy readings

Tonight my Inverter seems to have provided flawed Production Energy readings which equaled zero. This lead to a drop in the ConsumptionActiveEnergy reading in InfluxDB. This had the consequence that my dashboard now reports 320 kWh produced for today (way too much) because it is based on the spread of the largest and smallest value in _sum/ConsumptionActiveEnergy.

My question: can we make Energy channels resilient to those flawed readings? That would mean to implement a check whether a reading is less than the previous reading and to ignore that by replacing the flawed reading with the previous value. Of course it needs to be configurable.

This is unfortunately not a trivial problem, and I believe it cannot really be fixed in a generic way.

1 Custom ElementToChannelConverter

One way to fix the problem is to a logic to the conversion from Modbus-Register to Channel. In this example I used a custom ElementToChannelConverter for that purpose:

				new FC3ReadRegistersTask(4004, Priority.HIGH, //
						m(SymmetricMeter.ChannelId.ACTIVE_POWER, new SignedWordElement(4004),
								SIGNED_POWER_CONVERTER_AND_INVERT)), //

followed by

	private static final ElementToChannelConverter SIGNED_POWER_CONVERTER_AND_INVERT = new ElementToChannelConverter(//
			value -> {
				if (value == null) {
					return null;
				}
				int intValue = (Short) value;
				if (intValue == -10_000) {
					return 0; // ignore '-10_000'
				}
				return intValue * -1; // invert
			}, //
			value -> value);

Source

You can use this approach if you simply want to ignore certain values - e.g. zero in your case.

2 Additional Channel + OnChange listener

Second option is to introduce a second channel with a custom On-Change listener, like in this example:

		ORIGINAL_SOC(new IntegerDoc()//
				.onInit(channel -> { //
					final EvictingQueue<Integer> lastSocValues = EvictingQueue.create(100);
					((IntegerReadChannel) channel).onChange((oldValue, newValue) -> {
						Integer originalSocValue = newValue.get();
						Integer correctedSocValue = null;
						if (originalSocValue != null) {
							lastSocValues.add(originalSocValue);
							OptionalDouble averageSoc = lastSocValues.stream().mapToInt(Integer::intValue).average();
							if (averageSoc.isPresent()) {
								correctedSocValue = (int) averageSoc.getAsDouble();
							}
						}
						SymmetricEss parent = (SymmetricEss) channel.getComponent();
						parent._setSoc(correctedSocValue);
					});
				})),

Benefit of this approach is, that you get a nice onChange callback, that gives access to the previous and new value, so that you can easily filter whenever the next value is less.

This approach still does not fix the problem, when you restart OpenEMS during the period of the error.

3 Apply persisted value after restart

Third option is to make sure, that the new data is never less than the previous data - even on a restart of OpenEMS. We use this approach in the [CalculateEnergyFromPower](https://github.com/OpenEMS/openems/blob/develop/io.openems.edge.timedata.api/src/io/openems/edge/timedata/api/utils/CalculateEnergyFromPower.java) class, where we apply the new value only after validating it with the data loaded from a Timedata provider like RRD4j.

Thanks Stefan for the thoughtful reply. I decided to fix the issue in my Grafana Dashboard by filtering ‘outliers’ in my Flux queries. I do this by filtering anything out that has a negative difference or value is > 10 * standard deviation of series.

data = from(bucket: v.defaultBucket)
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: ® => r["_measurement"] == “data” and r["_field"] == “_sum/ConsumptionActiveEnergy”)
|> difference(nonNegative: true)

stddev = data
|> stddev()
|> findRecord(fn: (key) => key._measurement == “data”, idx: 0)

data
|> filter(fn: ® => r._value < 10.0 * stddev._value)
|> aggregateWindow(every: 1h, fn: sum, createEmpty: true)

Ok - fair enough. Also nice to see “Flux” being used. I have not touched it till now, but as InfluxDB version 2 was just released, I should maybe have a look…

Regarding the solution: I usually try to ‘fix’ these kind of errors as low-level as possible. For example Consumption data is used in Online-Monitoring and for prediction based control algorithms (like Awattar dynamic pricing integration), which rely on correct input data. Of course this only applies if you plan to use any of those.

After using the outlier detection for filtering values beyond 10 times standard deviation for one one day I reverted the change. In my case it accidentally dropped proper values.

I totally agree with you. If a meter or inverter Energy channel is behaving unexpected (no value returned at all, value is 0, value is declining instead of increasing then there needs to be a counter measure to fix this). This could be at the device/channel level then you need to do implemnt this for each and every supported device in the device specific logic.

Wouldn’t it be a better idea to implement this only once and ‘attach’ the logic only in the devices to the channels? Sounds like inheritance could be the way to do that.