Hello fellas,
slowly but surely I will come to a point where I run out of disk space for the Influxdb running on the server.
I never created additional retention policies besides the default autogen.
Can you tell me what RPs you set up? Any recommendations?
Hi Andreas,
I am using 30d> 15min, 365d> 60min for beside of autogen.
Maybe others have other values.
Regards,
Thorsten
Hi Thorsten,
thanks for your reply.
So I guess you created two CQs as described here https://docs.influxdata.com/influxdb/v1.7/guides/downsampling_and_retention/
to do the automatic downsample?
As I understand it, I will also have to change the queries in the Influxconnector inside OpenEMS to be able to query data outside of autogen.
That whole retention policy thing is a bit confusing to me. It seems that just creating additional RPs wouldn’t change anything unless creating CQs and therefore also changing the queries inside OpenEMS to be able to retrieve downsampled data.
Hi Andreas,
there are two related approaches.
Retention Policy
If you need to reduce the required disk space, this would be the way to go. As you suggested it might be required to define the retention policy that should be used in the query. Currently it is set only once here: https://github.com/OpenEMS/openems/blob/develop/io.openems.shared.influxdb/src/io/openems/shared/influxdb/InfluxConnector.java#L108
We are currently not using retention policies in production. Our database has about 2.5 TB at the moment, which is ‘not really’ an issue.
Continuous Queries
CQs can be used to downsample data which improves the speed on queries. E.g. as the OpenEMS UI by default queries 5 minute intervals, it might be a good idea to create CQs that precalculate this.
CQs are to queried like separate fields, so this would require adjusting the query accordingly e.g. here https://github.com/OpenEMS/openems/blob/e0e1e2a31238fede82f4f4a1e30a8666422180db/io.openems.shared.influxdb/src/io/openems/shared/influxdb/InfluxConnector.java#L441
The big downside of CQs in my opinion is, that it does not do automatic backfilling, i.e. if data is not stored in InfluxDB in real-time, it is not considered anymore for the CQ. As this can happen frequently with OpenEMS, e.g. if local internet connection is interrupted, this is a no-go for us. This is why we do not use CQs.
All in all I am currently not too happy with InfluxDB because of these issues. I hope that InfluxDB version 2 will solve them. I am also looking into TimescaleDB (https://www.timescale.com/) which seems to handle these cases much smarter, has built-in replication on server clusters and builds on rock-solid PostgresQL - but I did not start any development in that direction yet.
Regards,
Stefan
Hi Stefan,
thank you for the explenation. Now I have a much clearer understanding.