Internet DRAFT - draft-hongcs-t2trg-pem
draft-hongcs-t2trg-pem
T2TRG Hong, Choong Seon
Internet-Draft Kyung Hee University
Intended status: Standards Track Munir, Md. Shirajum
Expires: August 15, 2022 Kyung Hee University
Kitae Kim
Kyung Hee University
Seok Won Kang
Kyung Hee University
Proactive energy management for smart city with edge
computing using meta-reinforcement learning scheme
draft-hongcs-t2trg-pem-00
Abstract
Renewable energy enabled sustainable energy management ensures
a high degree of reliability in order to fulfill the energy
demand of a smart city. In such case, renewable energy
generation is random over time and also energy consumption of
smart city users’ is nondeterministic in nature. Therefore, to
ensure sustainable energy management for smart city,
a proactive energy management scheme should be integrated into
smart city network. In which, edge node should be considered as
local computational unit for each energy user and microgrid
controller should be played the role of energy management decision
aggregator. As a result, proactive energy management scheme
not only overcomes the challenges of renewable energy-aware
demand scheduling but also establishes a strong relationship
for both energy generation and consumption over time.
Therefore, a distributed mechanism is considered, where the edge
node for executing local agent to determine an individual
users’ policy with respect to energy consumption and renewable
energy generation (users’ own sources). On the other hand,
microgrid controller determines meta-policy through a meta-agent
with Recurrent Neural Network (RNN). Since a meta-agent accepts
local policy as an input with historical observations, which
ensures fast and efficient execution of proactive energy management
for the smart city.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as
"work in progress."
Hong, et al. Expires August 15, 2022 [Page 1]
Internet-Draft Proactive energy management for smart city October 2020
This Internet-Draft will expire on August xx, 20xx.
Copyright Notice
Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . .. . . . . . 2
1.1. Terminology and Requirements Language . . . . . . . . . 2
2. Energy Data Flow Management . . . . . . . . . . . . . . . . . . 3
2.1. Energy Data Flow . . . . . . . . . . . . . .. . . . . . 4
2.2. Energy Data Format . . . . . . . . . . . . . . . . . . . 5
3. Proactive Energy Management for Smart City . . . . . . . . . . 6
3.1 Meta-Reinforcement Learning with Edge Computing . . . . 7
3.2. Process flow of proactive energy management. . . . . . . .8
4. IANA Considerations . . . . . . .. . . . . . . . . . . . . . . 8
5. Security Considerations . . . . . . . . . . . . . . . . . . . 9
6. References . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
6.1. Normative References . . . . . . . . . . . . . . . . . . . . . 9
6.2. Informative References . . . . . .. . . . . . . . . . . . . . 9
Authors' Addresses . . . . . . . . . . . . . . . . . . . . .. . . . 10
Hong, et al. Expires August 15, 2022 [Page 2]
Internet-Draft Proactive energy management for smart city October 2020
1. Introduction
In the modern development arena, smart city, and renewable energy
are indispensable toward the ecological growth of urban technology
to enable sustainable smart services [a,b]. A microgrid is
capable to fulfill that huge amount of energy demand by enabling
the efficient demand scheduling of smart city energy consumptions.
However, the challenges come with the unpredictable nature of
both energy consumption and renewable generation, which also have
a strong relationship over the history of energy consumption and
generation [c,d].
Therefore, to overcome those challenges, a proactive energy
management is essential such that both energy consumption
and renewable energy generation can be considered. In order to
do that, meta-reinforcement learning (Meta-RL)-based [e]
energy scheduling model, in which this method is capable of
handling both energy consumption and generation with the historical
and current observations using Recurrent Neural Network (RNN).
Proactive energy management for the smart city should be solved by
distributed manner.
. First, a local agent with an edge computing facility determines
a local policy with respect to energy consumption and generation
(user’s own renewable sources) for its nearby energy users’.
. Second, by reusing the historical observations and local policy,
the microgrid controller estimates the meta-policy for energy
scheduling. Further, it takes necessary action based on meta-policy
to enable sustainable energy management for the smart city.
1.1. Terminology and Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
Hong, et al. Expires August 15, 2022 [Page 3]
Internet-Draft Proactive energy management for smart city October 2020
2. Energy Data Flow Management
The Microgrid-powered smart city includes both renewable and
non-renewable energy sources, where each individual building
has its own renewable (solar) energy sources. The city
connected with edge computing enabled wireless networks
to fulfill smart city services. Each energy user (i.e.,
home, building, school, commercial building, and so on)
is associated with its nearby edge computing server. Energy
demand, renewable generation, and required amount should send
to its associated edge server. Based on each user data, edge
server determine its own local energy management policy by
applying Deep Q-learning. The output (reward(t), action(t)
reward(t-1), action(t-1)) should send to microgrid controller.
Microgrid controller decides the overall energy management
policy for the smart city and send feed back to each user
via edge server. The communication should be through the wireless
communications protocol (LTE, 5G, LTU, Wifi, etc.) for exchanging
the energy management data.
Hong, et al. Expires August 15, 2022 [Page 4]
Internet-Draft Proactive energy management for smart city October 2020
2.1. Energy Data Flow
Energy data flow of proactive energy management for smart
city as shown in Figure 1.
. Energy user with renewable energy sources can send energy
demand and generation raw data to MEC Server # in smart
city network
. Each energy user's observational data (reward and action)
by executing reinforcement learning (Deep Q-learning) from
MEC should send to microgrid controller
. Other energy sources (except the energy sources that are
associated with smart city user) should send to microgrid
controller
. Microgrid controller send the energy management executing
command to each users’
Figure 1: Energy data flow of proactive energy management for
smart city
Hong, et al. Expires August 15, 2022 [Page 5]
Internet-Draft Proactive energy management for smart city October 2020
2.2 Energy Data Format
The data format complies with tuple.
Figure 2 represent the data format of proactive energy management
scheme.
Figure 2: Energy data format of proactive energy management
for smart city
Hong, et al. Expires August 15, 2022 [Page 6]
Internet-Draft Proactive energy management for smart city October 2020
3. Proactive Energy Management for Smart City
Each edge server estimates the local policy for the associated energy
users while Microgrid controller determines the meta policy using
a little amount of information from local policy. Establishing a
strong correlation between energy generation and consumption using
Markovian properties for each energy user.
Hong, et al. Expires August 15, 2022 [Page 7]
Internet-Draft Proactive energy management for smart city October 2020
3.1 Meta-Reinforcement Learning with Edge Computing
The proactive energy demand scheduling for smart city problem
is solved distributively, where first, obtain the local policy
by learning the local agent with respect to energy consumption
and renewable energy generation through the nearby edge server.
Second, in order to generate meta-policy, we send local policy
information to the microgrid controller alone with previous
policy observation, so that meta-agent can learn very fast
with the optimal decision. This procedure is the same for
every energy user and meta-RL model procedure is shown in
Figure 3.
Figure 3: Proactive energy management for smart city using
Meta-RL
Hong, et al. Expires August 15, 2022 [Page 8]
Internet-Draft Proactive energy management for smart city October 2020
3.2. Process flow of proactive energy management
Process flow of proactive energy management is illustrated in
Figure 4.
. Energy generation and demand data from all users at associated
edge server
. Local policy estimation process for each user’s at the edge server
using DQN and observation data by local agent send to microgrid
controller
. Meta energy management policy estimation using local policy at
microgrid controller and action command send to user through
edge server
. Apply energy management policy to smart city users by the edge
server service
Figure 4: Process flow of proactive energy management for smart city
Hong, et al. Expires August 15, 2022 [Page 9]
Internet-Draft Proactive energy management for smart city October 2020
4. IANA Considerations
There are no IANA considerations related to this document.
5. Security Considerations
This note touches communication security as in wireless communications
protocol (LTE, 5G, LTU, Wifi, etc.).
6. References
6.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[a] M. S. Munir, S. F. Abedin, M. G. R. Alam, N. H. Tran
and C. S. Hong, “Intelligent service fulfillment
for software defined networks in smart city,” 2018
International Conference on Information Networking
(ICOIN), pp. 516-521, 2018. (in Chiang Mai, Thailand).
[b] M. S. Munir, S. F. Abedin, M. G. R. Alam, D. H. Kim
and C. S. Hong, " Smart Agent based Dynamic Data
Aggregation for Delay Sensitive Smart City Services,"
Journal of KIISE, vol. 45, no. 4, pp. 395-402, April
2018.
[c] Y. Zhang, M. H. Hajiesmaili, S. Cai, M. Chen and Q. Zhu,
"Peak-Aware Online Economic Dispatching for Microgrids,"
in IEEE Transactions on Smart Grid, vol. 9, no. 1, pp.
323-335, Jan. 2018.
[D] M. S. Munir, S. F. Abedin, M. G. R. Alam, D. H. Kim,
and C. S. Hong, “RNN based Energy Demand Prediction for
Smart-Home in Smart-Grid Framework,” Korea Software
Congress 2017, pp. 437-439, 2017 (in Korea).
[E] J. X. Wang et al., “Learning to reinforcement learn,”
CogSci , 2017. (In London, UK).
6.2. Informative References
Hong, et al. Expires August 15, 2022 [Page 10]
Internet-Draft Proactive energy management for smart city October 2020
Authors' Addresses
Choong Seon Hong
Computer Science and Engineering Department, Kyung Hee University
Yongin, South Korea
Phone: +82 (0)31 201 2532
Email: cshong@khu.ac.kr
Md. Shirajum Munir
Computer Science and Engineering Department, Kyung Hee University
Yongin, South Korea
Phone: +82 (0)31 201 2987
Email: munir@khu.ac.kr
Ki Tae Kim
Computer Science and Engineering Department, Kyung Hee University
Yongin, South Korea
Phone: +82 (0)31 201 2532
Email: glideslope@khu.ac.kr
Seok Won Kang
Computer Science and Engineering Department, Kyung Hee University
Yongin, South Korea
Phone: +82 (0)31 201 2532
Email: dudtntdud@khu.ac.kr
