refactor: decouple db hooks from CFlatDB-based C*Manager objects, migrate to *Store structs (#5555)

## Motivation

As highlighted in https://github.com/dashpay/dash-issues/issues/52,
decoupling of `CFlatDB`-interacting components from managers of objects
like `CGovernanceManager` and `CSporkManager` is a key task for
achieving deglobalization of Dash-specific components.

The design of `CFlatDB` as a flat database agent relies on hooking into
the object's state its meant to load and store, using its
(de)serialization routines and other miscellaneous functions (notably,
without defining an interface) to achieve those ends. This approach was
taken predominantly for components that want a single-file cache.

Because of the method it uses to hook into the object (templates and the
use of temporary objects), it explicitly prevented passing arguments
into the object constructor, an explicit requirement for storing
references to other components during construction. This, in turn,
created an explicit dependency on those same components being available
in the global context, which would block the backport of bitcoin#21866,
a requirement for future backports meant to achieve parity in
`assumeutxo` support.

The design of these objects made no separation between persistent (i.e.
cached) and ephemeral (i.e. generated/fetched during initialization or
state transitions) data and the design of `CFlatDB` attempts to "clean"
the database by breaching this separation and attempting to access this
ephemeral data.

This might be acceptable if it is contained within the manager itself,
like `CSporkManager`'s `CheckAndRemove()` but is utterly unacceptable
when it relies on other managers (that, as a reminder, are only
accessible through the global state because of restrictions caused by
existing design), like `CGovernanceManager`'s `UpdateCachesAndClean()`.

This pull request aims to separate the `CFlatDB`-interacting portions of
these managers into a struct, with `CFlatDB` interacting only with this
struct, while the manager inherits the struct and manages
load/store/update of the database through the `CFlatDB` instance
initialized within its scope, though the instance only has knowledge of
what is exposed through the limited parent struct.

## Additional information

* As regards to existing behaviour, `CFlatDB` is written entirely as a
header as it relies on templates to specialize itself for the object it
hooks into. Attempting to split the logic and function definitions into
separate files will require you to explicitly define template
specializations, which is tedious.

* `m_db` is defined as a pointer as you cannot instantiate a
forward-declared template (see [this Stack Overflow
answer](https://stackoverflow.com/a/12797282) for more information),
which is done when defined as a member in the object scope.

* The conditional cache flush predicating on RPC _not_ being in the
warm-up state has been replaced with unconditional flushing of the
database on object destruction (@UdjinM6, is this acceptable?)

## TODOs

This is a list of things that aren't within the scope of this pull
request but should be addressed in subsequent pull requests

* [ ] Definition of an interface that `CFlatDB` stores are expected to
implement
* [ ] Lock annotations for all potential uses of members protected by
the `cs` mutex in each manager object and store
* [ ] Additional comments documenting what each function and member does
* [ ] Deglobalization of affected managers

---------

Co-authored-by: Kittywhiskers Van Gogh <63189531+kittywhiskers@users.noreply.github.com>

src/coinjoin/client.cpp

@@ -106,18 +106,18 @@ void CCoinJoinClientQueueManager::ProcessDSQueue(const CNode& peer, PeerManager&
                      dmn->pdmnState->addr.ToString());
             return;
         } else {
-            int64_t nLastDsq = mmetaman.GetMetaInfo(dmn->proTxHash)->GetLastDsq();
+            int64_t nLastDsq = mmetaman->GetMetaInfo(dmn->proTxHash)->GetLastDsq();
-            int64_t nDsqThreshold = mmetaman.GetDsqThreshold(dmn->proTxHash, mnList.GetValidMNsCount());
+            int64_t nDsqThreshold = mmetaman->GetDsqThreshold(dmn->proTxHash, mnList.GetValidMNsCount());
             LogPrint(BCLog::COINJOIN, "DSQUEUE -- nLastDsq: %d  nDsqThreshold: %d  nDsqCount: %d\n", nLastDsq,
-                     nDsqThreshold, mmetaman.GetDsqCount());
+                     nDsqThreshold, mmetaman->GetDsqCount());
             // don't allow a few nodes to dominate the queuing process
-            if (nLastDsq != 0 && nDsqThreshold > mmetaman.GetDsqCount()) {
+            if (nLastDsq != 0 && nDsqThreshold > mmetaman->GetDsqCount()) {
                 LogPrint(BCLog::COINJOIN, "DSQUEUE -- Masternode %s is sending too many dsq messages\n",
                          dmn->proTxHash.ToString());
                 return;
             }
 
-            mmetaman.AllowMixing(dmn->proTxHash);
+            mmetaman->AllowMixing(dmn->proTxHash);
 
             LogPrint(BCLog::COINJOIN, "DSQUEUE -- new CoinJoin queue (%s) from masternode %s\n", dsq.ToString(),
                      dmn->pdmnState->addr.ToString());
@@ -1144,13 +1144,13 @@ bool CCoinJoinClientSession::StartNewQueue(CAmount nBalanceNeedsAnonymized, CCon
             continue;
         }
 
-        int64_t nLastDsq = mmetaman.GetMetaInfo(dmn->proTxHash)->GetLastDsq();
+        int64_t nLastDsq = mmetaman->GetMetaInfo(dmn->proTxHash)->GetLastDsq();
-        int64_t nDsqThreshold = mmetaman.GetDsqThreshold(dmn->proTxHash, nMnCount);
+        int64_t nDsqThreshold = mmetaman->GetDsqThreshold(dmn->proTxHash, nMnCount);
-        if (nLastDsq != 0 && nDsqThreshold > mmetaman.GetDsqCount()) {
+        if (nLastDsq != 0 && nDsqThreshold > mmetaman->GetDsqCount()) {
             WalletCJLogPrint(m_wallet, "CCoinJoinClientSession::StartNewQueue -- Too early to mix on this masternode!" /* Continued */
                       " masternode=%s  addr=%s  nLastDsq=%d  nDsqThreshold=%d  nDsqCount=%d\n",
                 dmn->proTxHash.ToString(), dmn->pdmnState->addr.ToString(), nLastDsq,
-                nDsqThreshold, mmetaman.GetDsqCount());
+                nDsqThreshold, mmetaman->GetDsqCount());
             nTries++;
             continue;
         }

src/coinjoin/server.cpp

@@ -78,9 +78,9 @@ void CCoinJoinServer::ProcessDSACCEPT(CNode& peer, CDataStream& vRecv)
             }
         }
 
-        int64_t nLastDsq = mmetaman.GetMetaInfo(dmn->proTxHash)->GetLastDsq();
+        int64_t nLastDsq = mmetaman->GetMetaInfo(dmn->proTxHash)->GetLastDsq();
-        int64_t nDsqThreshold = mmetaman.GetDsqThreshold(dmn->proTxHash, mnList.GetValidMNsCount());
+        int64_t nDsqThreshold = mmetaman->GetDsqThreshold(dmn->proTxHash, mnList.GetValidMNsCount());
-        if (nLastDsq != 0 && nDsqThreshold > mmetaman.GetDsqCount()) {
+        if (nLastDsq != 0 && nDsqThreshold > mmetaman->GetDsqCount()) {
             if (fLogIPs) {
                 LogPrint(BCLog::COINJOIN, "DSACCEPT -- last dsq too recent, must wait: peer=%d, addr=%s\n", peer.GetId(), peer.addr.ToString());
             } else {
@@ -161,15 +161,15 @@ void CCoinJoinServer::ProcessDSQUEUE(const CNode& peer, PeerManager& peerman, CD
     }
 
     if (!dsq.fReady) {
-        int64_t nLastDsq = mmetaman.GetMetaInfo(dmn->proTxHash)->GetLastDsq();
+        int64_t nLastDsq = mmetaman->GetMetaInfo(dmn->proTxHash)->GetLastDsq();
-        int64_t nDsqThreshold = mmetaman.GetDsqThreshold(dmn->proTxHash, mnList.GetValidMNsCount());
+        int64_t nDsqThreshold = mmetaman->GetDsqThreshold(dmn->proTxHash, mnList.GetValidMNsCount());
-        LogPrint(BCLog::COINJOIN, "DSQUEUE -- nLastDsq: %d  nDsqThreshold: %d  nDsqCount: %d\n", nLastDsq, nDsqThreshold, mmetaman.GetDsqCount());
+        LogPrint(BCLog::COINJOIN, "DSQUEUE -- nLastDsq: %d  nDsqThreshold: %d  nDsqCount: %d\n", nLastDsq, nDsqThreshold, mmetaman->GetDsqCount());
         //don't allow a few nodes to dominate the queuing process
-        if (nLastDsq != 0 && nDsqThreshold > mmetaman.GetDsqCount()) {
+        if (nLastDsq != 0 && nDsqThreshold > mmetaman->GetDsqCount()) {
             LogPrint(BCLog::COINJOIN, "DSQUEUE -- Masternode %s is sending too many dsq messages\n", dmn->pdmnState->addr.ToString());
             return;
         }
-        mmetaman.AllowMixing(dmn->proTxHash);
+        mmetaman->AllowMixing(dmn->proTxHash);
 
         LogPrint(BCLog::COINJOIN, "DSQUEUE -- new CoinJoin queue (%s) from masternode %s\n", dsq.ToString(), dmn->pdmnState->addr.ToString());
 

src/evo/mnauth.cpp

@@ -123,7 +123,7 @@ void CMNAuth::ProcessMessage(CNode& peer, PeerManager& peerman, CConnman& connma
     }
 
     if (!peer.fInbound) {
-        mmetaman.GetMetaInfo(mnauth.proRegTxHash)->SetLastOutboundSuccess(GetAdjustedTime());
+        mmetaman->GetMetaInfo(mnauth.proRegTxHash)->SetLastOutboundSuccess(GetAdjustedTime());
         if (peer.m_masternode_probe_connection) {
             LogPrint(BCLog::NET_NETCONN, "CMNAuth::ProcessMessage -- Masternode probe successful for %s, disconnecting. peer=%d\n",
                      mnauth.proRegTxHash.ToString(), peer.GetId());

src/flat-database.h

@@ -6,6 +6,7 @@
 #define BITCOIN_FLAT_DATABASE_H
 
 #include <clientversion.h>
+#include <chainparams.h>
 #include <fs.h>
 #include <hash.h>
 #include <streams.h>
@@ -35,7 +36,7 @@ private:
     std::string strFilename;
     std::string strMagicMessage;
 
-    bool Write(const T& objToSave)
+    bool CoreWrite(const T& objToSave)
     {
         // LOCK(objToSave.cs);
 
@@ -70,7 +71,7 @@ private:
         return true;
     }
 
-    ReadResult Read(T& objToLoad, bool fDryRun = false)
+    ReadResult CoreRead(T& objToLoad)
     {
         //LOCK(objToLoad.cs);
 
@@ -151,28 +152,13 @@ private:
 
         LogPrintf("Loaded info from %s  %dms\n", strFilename, GetTimeMillis() - nStart);
         LogPrintf("     %s\n", objToLoad.ToString());
-        if(!fDryRun) {
-            LogPrintf("%s: Cleaning....\n", __func__);
-            objToLoad.CheckAndRemove();
-            LogPrintf("     %s\n", objToLoad.ToString());
-        }
 
         return Ok;
     }
 
-
+    bool Read(T& objToLoad)
-public:
-    CFlatDB(std::string strFilenameIn, std::string strMagicMessageIn)
     {
-        pathDB = GetDataDir() / strFilenameIn;
+        ReadResult readResult = CoreRead(objToLoad);
-        strFilename = strFilenameIn;
-        strMagicMessage = strMagicMessageIn;
-    }
-
-    bool Load(T& objToLoad)
-    {
-        LogPrintf("Reading info from %s...\n", strFilename);
-        ReadResult readResult = Read(objToLoad);
         if (readResult == FileError)
             LogPrintf("Missing file %s, will try to recreate\n", strFilename);
         else if (readResult != Ok)
@@ -191,36 +177,34 @@ public:
         return true;
     }
 
-    bool Dump(T& objToSave)
+public:
+    CFlatDB(std::string strFilenameIn, std::string strMagicMessageIn)
     {
-        int64_t nStart = GetTimeMillis();
+        pathDB = GetDataDir() / strFilenameIn;
+        strFilename = strFilenameIn;
+        strMagicMessage = strMagicMessageIn;
+    }
 
+    bool Load(T& objToLoad)
+    {
+        LogPrintf("Reading info from %s...\n", strFilename);
+        return Read(objToLoad);
+    }
+
+    bool Store(T& objToSave)
+    {
         LogPrintf("Verifying %s format...\n", strFilename);
         T tmpObjToLoad;
-        ReadResult readResult = Read(tmpObjToLoad, true);
+        if (!Read(tmpObjToLoad)) return false;
 
-        // there was an error and it was not an error on file opening => do not proceed
+        int64_t nStart = GetTimeMillis();
-        if (readResult == FileError)
-            LogPrintf("Missing file %s, will try to recreate\n", strFilename);
-        else if (readResult != Ok)
-        {
-            LogPrintf("Error reading %s: ", strFilename);
-            if(readResult == IncorrectFormat)
-                LogPrintf("%s: Magic is ok but data has invalid format, will try to recreate\n", __func__);
-            else
-            {
-                LogPrintf("%s: File format is unknown or invalid, please fix it manually\n", __func__);
-                return false;
-            }
-        }
 
         LogPrintf("Writing info to %s...\n", strFilename);
-        Write(objToSave);
+        CoreWrite(objToSave);
         LogPrintf("%s dump finished  %dms\n", strFilename, GetTimeMillis() - nStart);
 
         return true;
     }
-
 };
 
 

src/governance/governance.cpp

@@ -9,6 +9,7 @@
 #include <chainparams.h>
 #include <consensus/validation.h>
 #include <evo/deterministicmns.h>
+#include <flat-database.h>
 #include <governance/classes.h>
 #include <governance/validators.h>
 #include <masternode/meta.h>
@@ -26,27 +27,49 @@ std::unique_ptr<CGovernanceManager> governance;
 
 int nSubmittedFinalBudget;
 
-const std::string CGovernanceManager::SERIALIZATION_VERSION_STRING = "CGovernanceManager-Version-16";
+const std::string GovernanceStore::SERIALIZATION_VERSION_STRING = "CGovernanceManager-Version-16";
 const int CGovernanceManager::MAX_TIME_FUTURE_DEVIATION = 60 * 60;
 const int CGovernanceManager::RELIABLE_PROPAGATION_TIME = 60;
 
-CGovernanceManager::CGovernanceManager() :
+GovernanceStore::GovernanceStore() :
-    nTimeLastDiff(0),
+    cs(),
-    nCachedBlockHeight(0),
     mapObjects(),
     mapErasedGovernanceObjects(),
     cmapVoteToObject(MAX_CACHE_SIZE),
     cmapInvalidVotes(MAX_CACHE_SIZE),
     cmmapOrphanVotes(MAX_CACHE_SIZE),
     mapLastMasternodeObject(),
+    lastMNListForVotingKeys(std::make_shared<CDeterministicMNList>())
+{
+}
+
+CGovernanceManager::CGovernanceManager() :
+    m_db{std::make_unique<db_type>("governance.dat", "magicGovernanceCache")},
+    nTimeLastDiff(0),
+    nCachedBlockHeight(0),
     setRequestedObjects(),
     fRateChecksEnabled(true),
-    lastMNListForVotingKeys(std::make_shared<CDeterministicMNList>()),
+    votedFundingYesTriggerHash(std::nullopt)
-    votedFundingYesTriggerHash(std::nullopt),
-    cs()
 {
 }
 
+CGovernanceManager::~CGovernanceManager()
+{
+    if (!is_valid) return;
+    m_db->Store(*this);
+}
+
+bool CGovernanceManager::LoadCache(bool load_cache)
+{
+    assert(m_db != nullptr);
+    is_valid = load_cache ? m_db->Load(*this) : m_db->Store(*this);
+    if (is_valid && load_cache) {
+        CheckAndRemove();
+        InitOnLoad();
+    }
+    return is_valid;
+}
+
 // Accessors for thread-safe access to maps
 bool CGovernanceManager::HaveObjectForHash(const uint256& nHash) const
 {
@@ -318,7 +341,7 @@ void CGovernanceManager::UpdateCachesAndClean()
 
     LogPrint(BCLog::GOBJECT, "CGovernanceManager::UpdateCachesAndClean\n");
 
-    std::vector<uint256> vecDirtyHashes = mmetaman.GetAndClearDirtyGovernanceObjectHashes();
+    std::vector<uint256> vecDirtyHashes = mmetaman->GetAndClearDirtyGovernanceObjectHashes();
 
     LOCK2(cs_main, cs);
 
@@ -368,7 +391,7 @@ void CGovernanceManager::UpdateCachesAndClean()
         if ((pObj->IsSetCachedDelete() || pObj->IsSetExpired()) &&
             (nTimeSinceDeletion >= GOVERNANCE_DELETION_DELAY)) {
             LogPrint(BCLog::GOBJECT, "CGovernanceManager::UpdateCachesAndClean -- erase obj %s\n", (*it).first.ToString());
-            mmetaman.RemoveGovernanceObject(pObj->GetHash());
+            mmetaman->RemoveGovernanceObject(pObj->GetHash());
 
             // Remove vote references
             const object_ref_cm_t::list_t& listItems = cmapVoteToObject.GetItemList();
@@ -850,13 +873,13 @@ void CGovernanceManager::SyncObjects(CNode& peer, PeerManager& peerman, CConnman
     // do not provide any data until our node is synced
     if (!::masternodeSync->IsSynced()) return;
 
-    if (netfulfilledman.HasFulfilledRequest(peer.addr, NetMsgType::MNGOVERNANCESYNC)) {
+    if (netfulfilledman->HasFulfilledRequest(peer.addr, NetMsgType::MNGOVERNANCESYNC)) {
         // Asking for the whole list multiple times in a short period of time is no good
         LogPrint(BCLog::GOBJECT, "CGovernanceManager::%s -- peer already asked me for the list\n", __func__);
         peerman.Misbehaving(peer.GetId(), 20);
         return;
     }
-    netfulfilledman.AddFulfilledRequest(peer.addr, NetMsgType::MNGOVERNANCESYNC);
+    netfulfilledman->AddFulfilledRequest(peer.addr, NetMsgType::MNGOVERNANCESYNC);
 
     int nObjCount = 0;
 
@@ -1323,7 +1346,7 @@ void CGovernanceManager::InitOnLoad()
     LogPrintf("     %s\n", ToString());
 }
 
-std::string CGovernanceManager::ToString() const
+std::string GovernanceStore::ToString() const
 {
     LOCK(cs);
 

src/governance/governance.h

@@ -13,6 +13,8 @@
 
 class CBloomFilter;
 class CBlockIndex;
+template<typename T>
+class CFlatDB;
 class CInv;
 
 class CGovernanceManager;
@@ -124,14 +126,9 @@ public:
     }
 };
 
-//
+class GovernanceStore
-// Governance Manager : Contains all proposals for the budget
-//
-class CGovernanceManager
 {
-    friend class CGovernanceObject;
+protected:
-
-public: // Types
     struct last_object_rec {
         explicit last_object_rec(bool fStatusOKIn = true) :
             triggerBuffer(),
@@ -148,57 +145,95 @@ public: // Types
         bool fStatusOK;
     };
 
-
     using object_ref_cm_t = CacheMap<uint256, CGovernanceObject*>;
-
+    using txout_m_t = std::map<COutPoint, last_object_rec>;
     using vote_cmm_t = CacheMultiMap<uint256, vote_time_pair_t>;
 
-    using txout_m_t = std::map<COutPoint, last_object_rec>;
+protected:
-
-    using hash_s_t = std::set<uint256>;
-
-private:
     static constexpr int MAX_CACHE_SIZE = 1000000;
-
     static const std::string SERIALIZATION_VERSION_STRING;
 
-    static const int MAX_TIME_FUTURE_DEVIATION;
+public:
-    static const int RELIABLE_PROPAGATION_TIME;
+    // critical section to protect the inner data structures
-
+    mutable RecursiveMutex cs;
-    int64_t nTimeLastDiff;
-
-    // keep track of current block height
-    int nCachedBlockHeight;
 
+protected:
     // keep track of the scanning errors
     std::map<uint256, CGovernanceObject> mapObjects GUARDED_BY(cs);
-
     // mapErasedGovernanceObjects contains key-value pairs, where
     //   key   - governance object's hash
     //   value - expiration time for deleted objects
     std::map<uint256, int64_t> mapErasedGovernanceObjects;
-
-    std::map<uint256, CGovernanceObject> mapPostponedObjects;
-    hash_s_t setAdditionalRelayObjects;
-
     object_ref_cm_t cmapVoteToObject;
-
     CacheMap<uint256, CGovernanceVote> cmapInvalidVotes;
-
     vote_cmm_t cmmapOrphanVotes;
-
     txout_m_t mapLastMasternodeObject;
-
-    hash_s_t setRequestedObjects;
-
-    hash_s_t setRequestedVotes;
-
-    bool fRateChecksEnabled;
-
     // used to check for changed voting keys
     CDeterministicMNListPtr lastMNListForVotingKeys;
 
-    std::optional<uint256> votedFundingYesTriggerHash;
+public:
+    GovernanceStore();
+    ~GovernanceStore() = default;
+
+    template<typename Stream>
+    void Serialize(Stream &s) const
+    {
+        LOCK(cs);
+        s   << SERIALIZATION_VERSION_STRING
+            << mapErasedGovernanceObjects
+            << cmapInvalidVotes
+            << cmmapOrphanVotes
+            << mapObjects
+            << mapLastMasternodeObject
+            << *lastMNListForVotingKeys;
+    }
+
+    template<typename Stream>
+    void Unserialize(Stream &s)
+    {
+        Clear();
+
+        LOCK(cs);
+        std::string strVersion;
+        s >> strVersion;
+        if (strVersion != SERIALIZATION_VERSION_STRING) {
+            return;
+        }
+
+        s   >> mapErasedGovernanceObjects
+            >> cmapInvalidVotes
+            >> cmmapOrphanVotes
+            >> mapObjects
+            >> mapLastMasternodeObject
+            >> *lastMNListForVotingKeys;
+    }
+
+    void Clear()
+    {
+        LOCK(cs);
+
+        LogPrint(BCLog::GOBJECT, "Governance object manager was cleared\n");
+        mapObjects.clear();
+        mapErasedGovernanceObjects.clear();
+        cmapVoteToObject.Clear();
+        cmapInvalidVotes.Clear();
+        cmmapOrphanVotes.Clear();
+        mapLastMasternodeObject.clear();
+    }
+
+    std::string ToString() const;
+};
+
+//
+// Governance Manager : Contains all proposals for the budget
+//
+class CGovernanceManager : public GovernanceStore
+{
+    friend class CGovernanceObject;
+
+private:
+    using hash_s_t = std::set<uint256>;
+    using db_type = CFlatDB<GovernanceStore>;
 
     class ScopedLockBool
     {
@@ -220,13 +255,31 @@ private:
         }
     };
 
+private:
+    static const int MAX_TIME_FUTURE_DEVIATION;
+    static const int RELIABLE_PROPAGATION_TIME;
+
+private:
+    const std::unique_ptr<db_type> m_db;
+    bool is_valid{false};
+
+    int64_t nTimeLastDiff;
+    // keep track of current block height
+    int nCachedBlockHeight;
+    std::map<uint256, CGovernanceObject> mapPostponedObjects;
+    hash_s_t setAdditionalRelayObjects;
+    hash_s_t setRequestedObjects;
+    hash_s_t setRequestedVotes;
+    bool fRateChecksEnabled;
+    std::optional<uint256> votedFundingYesTriggerHash;
+
 public:
-    // critical section to protect the inner data structures
-    mutable RecursiveMutex cs;
-
     CGovernanceManager();
+    ~CGovernanceManager();
 
-    virtual ~CGovernanceManager() = default;
+    bool LoadCache(bool load_cache);
+
+    bool IsValid() const { return is_valid; }
 
     /**
      * This is called by AlreadyHave in net_processing.cpp as part of the inventory
@@ -263,55 +316,8 @@ public:
 
     void CheckAndRemove() { UpdateCachesAndClean(); }
 
-    void Clear()
-    {
-        LOCK(cs);
-
-        LogPrint(BCLog::GOBJECT, "Governance object manager was cleared\n");
-        mapObjects.clear();
-        mapErasedGovernanceObjects.clear();
-        cmapVoteToObject.Clear();
-        cmapInvalidVotes.Clear();
-        cmmapOrphanVotes.Clear();
-        mapLastMasternodeObject.clear();
-    }
-
-    std::string ToString() const;
     UniValue ToJson() const;
 
-    template<typename Stream>
-    void Serialize(Stream &s) const
-    {
-        LOCK(cs);
-        s   << SERIALIZATION_VERSION_STRING
-            << mapErasedGovernanceObjects
-            << cmapInvalidVotes
-            << cmmapOrphanVotes
-            << mapObjects
-            << mapLastMasternodeObject
-            << *lastMNListForVotingKeys;
-    }
-
-    template<typename Stream>
-    void Unserialize(Stream &s)
-    {
-        LOCK(cs);
-        Clear();
-
-        std::string strVersion;
-        s >> strVersion;
-        if (strVersion != SERIALIZATION_VERSION_STRING) {
-            return;
-        }
-
-        s   >> mapErasedGovernanceObjects
-            >> cmapInvalidVotes
-            >> cmmapOrphanVotes
-            >> mapObjects
-            >> mapLastMasternodeObject
-            >> *lastMNListForVotingKeys;
-    }
-
     void UpdatedBlockTip(const CBlockIndex* pindex, CConnman& connman);
     int64_t GetLastDiffTime() const { return nTimeLastDiff; }
     void UpdateLastDiffTime(int64_t nTimeIn) { nTimeLastDiff = nTimeIn; }

src/governance/object.cpp

@@ -199,7 +199,7 @@ bool CGovernanceObject::ProcessVote(const CGovernanceVote& vote, CGovernanceExce
         return false;
     }
 
-    if (!mmetaman.AddGovernanceVote(dmn->proTxHash, vote.GetParentHash())) {
+    if (!mmetaman->AddGovernanceVote(dmn->proTxHash, vote.GetParentHash())) {
         std::ostringstream ostr;
         ostr << "CGovernanceObject::ProcessVote -- Unable to add governance vote"
              << ", MN outpoint = " << vote.GetMasternodeOutpoint().ToStringShort()

src/init.cpp

@@ -237,11 +237,6 @@ void PrepareShutdown(NodeContext& node)
     StopHTTPServer();
     if (node.llmq_ctx) node.llmq_ctx->Stop();
 
-    // fRPCInWarmup should be `false` if we completed the loading sequence
-    // before a shutdown request was received
-    std::string statusmessage;
-    bool fRPCInWarmup = RPCIsInWarmup(&statusmessage);
-
     for (const auto& client : node.chain_clients) {
         client->flush();
     }
@@ -277,20 +272,6 @@ void PrepareShutdown(NodeContext& node)
     // CValidationInterface callbacks, flush them...
     GetMainSignals().FlushBackgroundCallbacks();
 
-    if (!fRPCInWarmup) {
-        // STORE DATA CACHES INTO SERIALIZED DAT FILES
-        CFlatDB<CMasternodeMetaMan> flatdb1("mncache.dat", "magicMasternodeCache");
-        flatdb1.Dump(mmetaman);
-        CFlatDB<CNetFulfilledRequestManager> flatdb4("netfulfilled.dat", "magicFulfilledCache");
-        flatdb4.Dump(netfulfilledman);
-        CFlatDB<CSporkManager> flatdb6("sporks.dat", "magicSporkCache");
-        flatdb6.Dump(*::sporkManager);
-        if (!fDisableGovernance) {
-            CFlatDB<CGovernanceManager> flatdb3("governance.dat", "magicGovernanceCache");
-            flatdb3.Dump(*::governance);
-        }
-    }
-
     // After the threads that potentially access these pointers have been stopped,
     // destruct and reset all to nullptr.
     node.peerman.reset();
@@ -324,6 +305,8 @@ void PrepareShutdown(NodeContext& node)
     ::governance.reset();
     ::sporkManager.reset();
     ::masternodeSync.reset();
+    ::netfulfilledman.reset();
+    ::mmetaman.reset();
 
     // Stop and delete all indexes only after flushing background callbacks.
     if (g_txindex) {
@@ -1696,6 +1679,7 @@ bool AppInitMain(const CoreContext& context, NodeContext& node, interfaces::Bloc
     node.chainman = &g_chainman;
     ChainstateManager& chainman = *Assert(node.chainman);
 
+    assert(!::governance);
     ::governance = std::make_unique<CGovernanceManager>();
 
     assert(!node.peerman);
@@ -1706,7 +1690,6 @@ bool AppInitMain(const CoreContext& context, NodeContext& node, interfaces::Bloc
 
     assert(!::sporkManager);
     ::sporkManager = std::make_unique<CSporkManager>();
-    ::masternodeSync = std::make_unique<CMasternodeSync>(*node.connman, *::governance);
 
     std::vector<std::string> vSporkAddresses;
     if (args.IsArgSet("-sporkaddr")) {
@@ -1732,6 +1715,8 @@ bool AppInitMain(const CoreContext& context, NodeContext& node, interfaces::Bloc
         }
     }
 
+    ::masternodeSync = std::make_unique<CMasternodeSync>(*node.connman, *::governance);
+
     // sanitize comments per BIP-0014, format user agent and check total size
     std::vector<std::string> uacomments;
 
@@ -1852,8 +1837,8 @@ bool AppInitMain(const CoreContext& context, NodeContext& node, interfaces::Bloc
     }
 #endif
 
-    assert(masternodeSync != nullptr);
+    assert(::governance != nullptr);
-    assert(governance != nullptr);
+    assert(::masternodeSync != nullptr);
     pdsNotificationInterface = new CDSNotificationInterface(
         *node.connman, *::masternodeSync, ::deterministicMNManager, *::governance, node.llmq_ctx, node.cj_ctx
     );
@@ -1867,10 +1852,9 @@ bool AppInitMain(const CoreContext& context, NodeContext& node, interfaces::Bloc
 
     // ********************************************************* Step 7a: Load sporks
 
-    uiInterface.InitMessage(_("Loading sporks cache...").translated);
+    if (!::sporkManager->LoadCache()) {
-    CFlatDB<CSporkManager> flatdb6("sporks.dat", "magicSporkCache");
+        auto file_path = (GetDataDir() / "sporks.dat").string();
-    if (!flatdb6.Load(*::sporkManager)) {
+        return InitError(strprintf(_("Failed to load sporks cache from %s"), file_path));
-        return InitError(strprintf(_("Failed to load sporks cache from %s"), (GetDataDir() / "sporks.dat").string()));
     }
 
     // ********************************************************* Step 7b: load block chain
@@ -2202,6 +2186,40 @@ bool AppInitMain(const CoreContext& context, NodeContext& node, interfaces::Bloc
     g_wallet_init_interface.InitCoinJoinSettings(*node.cj_ctx->clientman);
 #endif // ENABLE_WALLET
 
+    // ********************************************************* Step 7d: Setup other Dash services
+
+    bool fLoadCacheFiles = !(fReindex || fReindexChainState) && (::ChainActive().Tip() != nullptr);
+
+    if (!fDisableGovernance) {
+        if (!::governance->LoadCache(fLoadCacheFiles)) {
+            auto file_path = (GetDataDir() / "governance.dat").string();
+            if (fLoadCacheFiles && !fDisableGovernance) {
+                return InitError(strprintf(_("Failed to load governance cache from %s"), file_path));
+            }
+            return InitError(strprintf(_("Failed to clear governance cache at %s"), file_path));
+        }
+    }
+
+    assert(!::mmetaman);
+    ::mmetaman = std::make_unique<CMasternodeMetaMan>(fLoadCacheFiles);
+    if (!::mmetaman->IsValid()) {
+        auto file_path = (GetDataDir() / "mncache.dat").string();
+        if (fLoadCacheFiles) {
+            return InitError(strprintf(_("Failed to load masternode cache from %s"), file_path));
+        }
+        return InitError(strprintf(_("Failed to clear masternode cache at %s"), file_path));
+    }
+
+    assert(!::netfulfilledman);
+    ::netfulfilledman = std::make_unique<CNetFulfilledRequestManager>(fLoadCacheFiles);
+    if (!::netfulfilledman->IsValid()) {
+        auto file_path = (GetDataDir() / "netfulfilled.dat").string();
+        if (fLoadCacheFiles) {
+            return InitError(strprintf(_("Failed to load fulfilled requests cache from %s"), file_path));
+        }
+        return InitError(strprintf(_("Failed to clear fulfilled requests cache at %s"), file_path));
+    }
+
     // ********************************************************* Step 8: start indexers
     if (args.GetBoolArg("-txindex", DEFAULT_TXINDEX)) {
         g_txindex = std::make_unique<TxIndex>(nTxIndexCache, false, fReindex);
@@ -2263,60 +2281,9 @@ bool AppInitMain(const CoreContext& context, NodeContext& node, interfaces::Bloc
         return false;
     }
 
-    // ********************************************************* Step 10a: Load cache data
+    // ********************************************************* Step 10a: schedule Dash-specific tasks
 
-    // LOAD SERIALIZED DAT FILES INTO DATA CACHES FOR INTERNAL USE
+    node.scheduler->scheduleEvery(std::bind(&CNetFulfilledRequestManager::DoMaintenance, std::ref(*netfulfilledman)), std::chrono::minutes{1});
-
-    bool fLoadCacheFiles = !(fReindex || fReindexChainState) && (::ChainActive().Tip() != nullptr);
-    fs::path pathDB = GetDataDir();
-    std::string strDBName;
-
-    strDBName = "mncache.dat";
-    uiInterface.InitMessage(_("Loading masternode cache...").translated);
-    CFlatDB<CMasternodeMetaMan> flatdb1(strDBName, "magicMasternodeCache");
-    if (fLoadCacheFiles) {
-        if(!flatdb1.Load(mmetaman)) {
-            return InitError(strprintf(_("Failed to load masternode cache from %s"), (pathDB / strDBName).string()));
-        }
-    } else {
-        CMasternodeMetaMan mmetamanTmp;
-        if(!flatdb1.Dump(mmetamanTmp)) {
-            return InitError(strprintf(_("Failed to clear masternode cache at %s"), (pathDB / strDBName).string()));
-        }
-    }
-
-    strDBName = "governance.dat";
-    uiInterface.InitMessage(_("Loading governance cache...").translated);
-    CFlatDB<CGovernanceManager> flatdb3(strDBName, "magicGovernanceCache");
-    if (fLoadCacheFiles && !fDisableGovernance) {
-        if(!flatdb3.Load(*::governance)) {
-            return InitError(strprintf(_("Failed to load governance cache from %s"), (pathDB / strDBName).string()));
-        }
-        ::governance->InitOnLoad();
-    } else {
-        CGovernanceManager governanceTmp;
-        if(!flatdb3.Dump(governanceTmp)) {
-            return InitError(strprintf(_("Failed to clear governance cache at %s"), (pathDB / strDBName).string()));
-        }
-    }
-
-    strDBName = "netfulfilled.dat";
-    uiInterface.InitMessage(_("Loading fulfilled requests cache...").translated);
-    CFlatDB<CNetFulfilledRequestManager> flatdb4(strDBName, "magicFulfilledCache");
-    if (fLoadCacheFiles) {
-        if(!flatdb4.Load(netfulfilledman)) {
-            return InitError(strprintf(_("Failed to load fulfilled requests cache from %s"),(pathDB / strDBName).string()));
-        }
-    } else {
-        CNetFulfilledRequestManager netfulfilledmanTmp;
-        if(!flatdb4.Dump(netfulfilledmanTmp)) {
-            return InitError(strprintf(_("Failed to clear fulfilled requests cache at %s"),(pathDB / strDBName).string()));
-        }
-    }
-
-    // ********************************************************* Step 10b: schedule Dash-specific tasks
-
-    node.scheduler->scheduleEvery(std::bind(&CNetFulfilledRequestManager::DoMaintenance, std::ref(netfulfilledman)), std::chrono::minutes{1});
     node.scheduler->scheduleEvery(std::bind(&CMasternodeSync::DoMaintenance, std::ref(*::masternodeSync)), std::chrono::seconds{1});
     node.scheduler->scheduleEvery(std::bind(&CMasternodeUtils::DoMaintenance, std::ref(*node.connman), std::ref(*::masternodeSync), std::ref(*node.cj_ctx)), std::chrono::minutes{1});
     node.scheduler->scheduleEvery(std::bind(&CDeterministicMNManager::DoMaintenance, std::ref(*deterministicMNManager)), std::chrono::seconds{10});

src/llmq/dkgsession.cpp

@@ -463,7 +463,7 @@ void CDKGSession::VerifyConnectionAndMinProtoVersions() const
             logger.Batch("%s does not have min proto version %d (has %d)", m->dmn->proTxHash.ToString(), MIN_MASTERNODE_PROTO_VERSION, it->second);
         }
 
-        if (mmetaman.GetMetaInfo(m->dmn->proTxHash)->OutboundFailedTooManyTimes()) {
+        if (mmetaman->GetMetaInfo(m->dmn->proTxHash)->OutboundFailedTooManyTimes()) {
             m->badConnection = true;
             logger.Batch("%s failed to connect to it too many times", m->dmn->proTxHash.ToString());
         }

src/llmq/utils.cpp

@@ -938,7 +938,7 @@ void AddQuorumProbeConnections(const Consensus::LLMQParams& llmqParams, const CB
         if (dmn->proTxHash == myProTxHash) {
             continue;
         }
-        auto lastOutbound = mmetaman.GetMetaInfo(dmn->proTxHash)->GetLastOutboundSuccess();
+        auto lastOutbound = mmetaman->GetMetaInfo(dmn->proTxHash)->GetLastOutboundSuccess();
         if (curTime - lastOutbound < 10 * 60) {
             // avoid re-probing nodes too often
             continue;

src/masternode/meta.cpp

@@ -4,13 +4,31 @@
 
 #include <masternode/meta.h>
 
+#include <flat-database.h>
 #include <timedata.h>
 
 #include <sstream>
 
-CMasternodeMetaMan mmetaman;
+std::unique_ptr<CMasternodeMetaMan> mmetaman;
 
-const std::string CMasternodeMetaMan::SERIALIZATION_VERSION_STRING = "CMasternodeMetaMan-Version-3";
+const std::string MasternodeMetaStore::SERIALIZATION_VERSION_STRING = "CMasternodeMetaMan-Version-3";
+
+CMasternodeMetaMan::CMasternodeMetaMan(bool load_cache) :
+    m_db{std::make_unique<db_type>("mncache.dat", "magicMasternodeCache")},
+    is_valid{
+        [&]() -> bool {
+            assert(m_db != nullptr);
+            return load_cache ? m_db->Load(*this) : m_db->Store(*this);
+        }()
+    }
+{
+}
+
+CMasternodeMetaMan::~CMasternodeMetaMan()
+{
+    if (!is_valid) return;
+    m_db->Store(*this);
+}
 
 UniValue CMasternodeMetaInfo::ToJson() const
 {
@@ -110,14 +128,7 @@ std::vector<uint256> CMasternodeMetaMan::GetAndClearDirtyGovernanceObjectHashes(
     return vecTmp;
 }
 
-void CMasternodeMetaMan::Clear()
+std::string MasternodeMetaStore::ToString() const
-{
-    LOCK(cs);
-    metaInfos.clear();
-    vecDirtyGovernanceObjectHashes.clear();
-}
-
-std::string CMasternodeMetaMan::ToString() const
 {
     std::ostringstream info;
     LOCK(cs);

src/masternode/meta.h

@@ -6,14 +6,17 @@
 #define BITCOIN_MASTERNODE_META_H
 
 #include <serialize.h>
+#include <sync.h>
+#include <uint256.h>
 
 #include <univalue.h>
 
 #include <atomic>
-#include <uint256.h>
+#include <memory>
-#include <sync.h>
 
 class CConnman;
+template<typename T>
+class CFlatDB;
 
 static constexpr int MASTERNODE_MAX_MIXING_TXES{5};
 static constexpr int MASTERNODE_MAX_FAILED_OUTBOUND_ATTEMPTS{5};
@@ -89,16 +92,13 @@ public:
 };
 using CMasternodeMetaInfoPtr = std::shared_ptr<CMasternodeMetaInfo>;
 
-class CMasternodeMetaMan
+class MasternodeMetaStore
 {
-private:
+protected:
     static const std::string SERIALIZATION_VERSION_STRING;
 
     mutable RecursiveMutex cs;
-
     std::map<uint256, CMasternodeMetaInfoPtr> metaInfos GUARDED_BY(cs);
-    std::vector<uint256> vecDirtyGovernanceObjectHashes GUARDED_BY(cs);
-
     // keep track of dsq count to prevent masternodes from gaming coinjoin queue
     std::atomic<int64_t> nDsqCount{0};
 
@@ -117,8 +117,9 @@ public:
     template<typename Stream>
     void Unserialize(Stream &s)
     {
-        LOCK(cs);
         Clear();
+
+        LOCK(cs);
         std::string strVersion;
         s >> strVersion;
         if (strVersion != SERIALIZATION_VERSION_STRING) {
@@ -132,7 +133,33 @@ public:
         }
     }
 
+    void Clear()
+    {
+        LOCK(cs);
+
+        metaInfos.clear();
+    }
+
+    std::string ToString() const;
+};
+
+class CMasternodeMetaMan : public MasternodeMetaStore
+{
+private:
+    using db_type = CFlatDB<MasternodeMetaStore>;
+
+private:
+    const std::unique_ptr<db_type> m_db;
+    const bool is_valid{false};
+
+    std::vector<uint256> vecDirtyGovernanceObjectHashes GUARDED_BY(cs);
+
 public:
+    explicit CMasternodeMetaMan(bool load_cache);
+    ~CMasternodeMetaMan();
+
+    bool IsValid() const { return is_valid; }
+
     CMasternodeMetaInfoPtr GetMetaInfo(const uint256& proTxHash, bool fCreate = true);
 
     int64_t GetDsqCount() const { return nDsqCount; }
@@ -145,14 +172,8 @@ public:
     void RemoveGovernanceObject(const uint256& nGovernanceObjectHash);
 
     std::vector<uint256> GetAndClearDirtyGovernanceObjectHashes();
-
-    void Clear();
-    // Needed to avoid errors in flat-database.h
-    void CheckAndRemove() const {};
-
-    std::string ToString() const;
 };
 
-extern CMasternodeMetaMan mmetaman;
+extern std::unique_ptr<CMasternodeMetaMan> mmetaman;
 
 #endif // BITCOIN_MASTERNODE_META_H

src/masternode/sync.cpp

@@ -76,7 +76,7 @@ void CMasternodeSync::SwitchToNextAsset()
             uiInterface.NotifyAdditionalDataSyncProgressChanged(1);
 
             connman.ForEachNode(CConnman::AllNodes, [](CNode* pnode) {
-                netfulfilledman.AddFulfilledRequest(pnode->addr, "full-sync");
+                netfulfilledman->AddFulfilledRequest(pnode->addr, "full-sync");
             });
             LogPrintf("CMasternodeSync::SwitchToNextAsset -- Sync has finished\n");
 
@@ -159,13 +159,13 @@ void CMasternodeSync::ProcessTick()
         if (!pnode->CanRelay() || (fMasternodeMode && pnode->fInbound)) continue;
 
         {
-            if ((pnode->HasPermission(PF_NOBAN) || pnode->m_manual_connection) && !netfulfilledman.HasFulfilledRequest(pnode->addr, strAllow)) {
+            if ((pnode->HasPermission(PF_NOBAN) || pnode->m_manual_connection) && !netfulfilledman->HasFulfilledRequest(pnode->addr, strAllow)) {
-                netfulfilledman.RemoveAllFulfilledRequests(pnode->addr);
+                netfulfilledman->RemoveAllFulfilledRequests(pnode->addr);
-                netfulfilledman.AddFulfilledRequest(pnode->addr, strAllow);
+                netfulfilledman->AddFulfilledRequest(pnode->addr, strAllow);
                 LogPrintf("CMasternodeSync::ProcessTick -- skipping mnsync restrictions for peer=%d\n", pnode->GetId());
             }
 
-            if(netfulfilledman.HasFulfilledRequest(pnode->addr, "full-sync")) {
+            if(netfulfilledman->HasFulfilledRequest(pnode->addr, "full-sync")) {
                 // We already fully synced from this node recently,
                 // disconnect to free this connection slot for another peer.
                 pnode->fDisconnect = true;
@@ -175,9 +175,9 @@ void CMasternodeSync::ProcessTick()
 
             // SPORK : ALWAYS ASK FOR SPORKS AS WE SYNC
 
-            if(!netfulfilledman.HasFulfilledRequest(pnode->addr, "spork-sync")) {
+            if(!netfulfilledman->HasFulfilledRequest(pnode->addr, "spork-sync")) {
                 // always get sporks first, only request once from each peer
-                netfulfilledman.AddFulfilledRequest(pnode->addr, "spork-sync");
+                netfulfilledman->AddFulfilledRequest(pnode->addr, "spork-sync");
                 // get current network sporks
                 connman.PushMessage(pnode, msgMaker.Make(NetMsgType::GETSPORKS));
                 LogPrint(BCLog::MNSYNC, "CMasternodeSync::ProcessTick -- nTick %d nCurrentAsset %d -- requesting sporks from peer=%d\n", nTick, nCurrentAsset, pnode->GetId());
@@ -201,9 +201,9 @@ void CMasternodeSync::ProcessTick()
                     if (gArgs.GetBoolArg("-syncmempool", DEFAULT_SYNC_MEMPOOL)) {
                         // Now that the blockchain is synced request the mempool from the connected outbound nodes if possible
                         for (auto pNodeTmp : vNodesCopy) {
-                            bool fRequestedEarlier = netfulfilledman.HasFulfilledRequest(pNodeTmp->addr, "mempool-sync");
+                            bool fRequestedEarlier = netfulfilledman->HasFulfilledRequest(pNodeTmp->addr, "mempool-sync");
                             if (pNodeTmp->nVersion >= 70216 && !pNodeTmp->fInbound && !fRequestedEarlier && !pNodeTmp->IsBlockRelayOnly()) {
-                                netfulfilledman.AddFulfilledRequest(pNodeTmp->addr, "mempool-sync");
+                                netfulfilledman->AddFulfilledRequest(pNodeTmp->addr, "mempool-sync");
                                 connman.PushMessage(pNodeTmp, msgMaker.Make(NetMsgType::MEMPOOL));
                                 LogPrint(BCLog::MNSYNC, "CMasternodeSync::ProcessTick -- nTick %d nCurrentAsset %d -- syncing mempool from peer=%d\n", nTick, nCurrentAsset, pNodeTmp->GetId());
                             }
@@ -235,12 +235,12 @@ void CMasternodeSync::ProcessTick()
                 }
 
                 // only request obj sync once from each peer
-                if(netfulfilledman.HasFulfilledRequest(pnode->addr, "governance-sync")) {
+                if(netfulfilledman->HasFulfilledRequest(pnode->addr, "governance-sync")) {
                     // will request votes on per-obj basis from each node in a separate loop below
                     // to avoid deadlocks here
                     continue;
                 }
-                netfulfilledman.AddFulfilledRequest(pnode->addr, "governance-sync");
+                netfulfilledman->AddFulfilledRequest(pnode->addr, "governance-sync");
 
                 nTriedPeerCount++;
 
@@ -260,7 +260,7 @@ void CMasternodeSync::ProcessTick()
 
     // request votes on per-obj basis from each node
     for (auto& pnode : vNodesCopy) {
-        if(!netfulfilledman.HasFulfilledRequest(pnode->addr, "governance-sync")) {
+        if(!netfulfilledman->HasFulfilledRequest(pnode->addr, "governance-sync")) {
             continue; // to early for this node
         }
         int nObjsLeftToAsk = m_govman.RequestGovernanceObjectVotes(*pnode, connman);

src/net.cpp

@@ -2513,6 +2513,8 @@ void CConnman::ThreadOpenMasternodeConnections()
     if (gArgs.IsArgSet("-connect") && gArgs.GetArgs("-connect").size() > 0)
         return;
 
+    assert(::mmetaman != nullptr);
+
     auto& chainParams = Params();
 
     bool didConnect = false;
@@ -2579,7 +2581,7 @@ void CConnman::ThreadOpenMasternodeConnections()
                         continue;
                     }
                     if (!connectedNodes.count(addr2) && !IsMasternodeOrDisconnectRequested(addr2) && !connectedProRegTxHashes.count(proRegTxHash)) {
-                        int64_t lastAttempt = mmetaman.GetMetaInfo(dmn->proTxHash)->GetLastOutboundAttempt();
+                        int64_t lastAttempt = mmetaman->GetMetaInfo(dmn->proTxHash)->GetLastOutboundAttempt();
                         // back off trying connecting to an address if we already tried recently
                         if (nANow - lastAttempt < chainParams.LLMQConnectionRetryTimeout()) {
                             continue;
@@ -2603,14 +2605,14 @@ void CConnman::ThreadOpenMasternodeConnections()
                 bool connectedAndOutbound = connectedProRegTxHashes.count(dmn->proTxHash) && !connectedProRegTxHashes[dmn->proTxHash];
                 if (connectedAndOutbound) {
                     // we already have an outbound connection to this MN so there is no theed to probe it again
-                    mmetaman.GetMetaInfo(dmn->proTxHash)->SetLastOutboundSuccess(nANow);
+                    mmetaman->GetMetaInfo(dmn->proTxHash)->SetLastOutboundSuccess(nANow);
                     it = masternodePendingProbes.erase(it);
                     continue;
                 }
 
                 ++it;
 
-                int64_t lastAttempt = mmetaman.GetMetaInfo(dmn->proTxHash)->GetLastOutboundAttempt();
+                int64_t lastAttempt = mmetaman->GetMetaInfo(dmn->proTxHash)->GetLastOutboundAttempt();
                 // back off trying connecting to an address if we already tried recently
                 if (nANow - lastAttempt < chainParams.LLMQConnectionRetryTimeout()) {
                     continue;
@@ -2661,7 +2663,7 @@ void CConnman::ThreadOpenMasternodeConnections()
 
         didConnect = true;
 
-        mmetaman.GetMetaInfo(connectToDmn->proTxHash)->SetLastOutboundAttempt(nANow);
+        mmetaman->GetMetaInfo(connectToDmn->proTxHash)->SetLastOutboundAttempt(nANow);
 
         OpenMasternodeConnection(CAddress(connectToDmn->pdmnState->addr, NODE_NETWORK), isProbe);
         // should be in the list now if connection was opened
@@ -2674,7 +2676,7 @@ void CConnman::ThreadOpenMasternodeConnections()
         if (!connected) {
             LogPrint(BCLog::NET_NETCONN, "CConnman::%s -- connection failed for masternode  %s, service=%s\n", __func__, connectToDmn->proTxHash.ToString(), connectToDmn->pdmnState->addr.ToString(false));
             // Will take a few consequent failed attempts to PoSe-punish a MN.
-            if (mmetaman.GetMetaInfo(connectToDmn->proTxHash)->OutboundFailedTooManyTimes()) {
+            if (mmetaman->GetMetaInfo(connectToDmn->proTxHash)->OutboundFailedTooManyTimes()) {
                 LogPrint(BCLog::NET_NETCONN, "CConnman::%s -- failed to connect to masternode %s too many times\n", __func__, connectToDmn->proTxHash.ToString());
             }
         }

src/net_processing.cpp

@@ -2731,6 +2731,8 @@ void PeerManagerImpl::ProcessGetCFCheckPt(CNode& peer, CDataStream& vRecv, const
 
 std::pair<bool /*ret*/, bool /*do_return*/> static ValidateDSTX(CTxMemPool& mempool, ChainstateManager& chainman, CCoinJoinBroadcastTx& dstx, uint256 hashTx)
 {
+    assert(::mmetaman != nullptr);
+
     if (!dstx.IsValidStructure()) {
         LogPrint(BCLog::COINJOIN, "DSTX -- Invalid DSTX structure: %s\n", hashTx.ToString());
         return {false, true};
@@ -2773,7 +2775,7 @@ std::pair<bool /*ret*/, bool /*do_return*/> static ValidateDSTX(CTxMemPool& memp
         return {false, true};
     }
 
-    if (!mmetaman.GetMetaInfo(dmn->proTxHash)->IsValidForMixingTxes()) {
+    if (!mmetaman->GetMetaInfo(dmn->proTxHash)->IsValidForMixingTxes()) {
         LogPrint(BCLog::COINJOIN, "DSTX -- Masternode %s is sending too many transactions %s\n", dstx.masternodeOutpoint.ToStringShort(), hashTx.ToString());
         return {true, true};
         // TODO: Not an error? Could it be that someone is relaying old DSTXes
@@ -2787,7 +2789,7 @@ std::pair<bool /*ret*/, bool /*do_return*/> static ValidateDSTX(CTxMemPool& memp
 
     LogPrint(BCLog::COINJOIN, "DSTX -- Got Masternode transaction %s\n", hashTx.ToString());
     mempool.PrioritiseTransaction(hashTx, 0.1*COIN);
-    mmetaman.DisallowMixing(dmn->proTxHash);
+    mmetaman->DisallowMixing(dmn->proTxHash);
 
     return {true, false};
 }

src/netfulfilledman.cpp

@@ -3,11 +3,32 @@
 // file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 #include <chainparams.h>
+#include <flat-database.h>
 #include <netfulfilledman.h>
 #include <shutdown.h>
 #include <util/system.h>
 
-CNetFulfilledRequestManager netfulfilledman;
+std::unique_ptr<CNetFulfilledRequestManager> netfulfilledman;
+
+CNetFulfilledRequestManager::CNetFulfilledRequestManager(bool load_cache) :
+    m_db{std::make_unique<db_type>("netfulfilled.dat", "magicFulfilledCache")},
+    is_valid{
+        [&]() -> bool {
+            assert(m_db != nullptr);
+            return load_cache ? m_db->Load(*this) : m_db->Store(*this);
+        }()
+    }
+{
+    if (is_valid && load_cache) {
+        CheckAndRemove();
+    }
+}
+
+CNetFulfilledRequestManager::~CNetFulfilledRequestManager()
+{
+    if (!is_valid) return;
+    m_db->Store(*this);
+}
 
 void CNetFulfilledRequestManager::AddFulfilledRequest(const CService& addr, const std::string& strRequest)
 {
@@ -62,13 +83,13 @@ void CNetFulfilledRequestManager::CheckAndRemove()
     }
 }
 
-void CNetFulfilledRequestManager::Clear()
+void NetFulfilledRequestStore::Clear()
 {
     LOCK(cs_mapFulfilledRequests);
     mapFulfilledRequests.clear();
 }
 
-std::string CNetFulfilledRequestManager::ToString() const
+std::string NetFulfilledRequestStore::ToString() const
 {
     std::ostringstream info;
     info << "Nodes with fulfilled requests: " << (int)mapFulfilledRequests.size();

src/netfulfilledman.h

@@ -9,41 +9,61 @@
 #include <serialize.h>
 #include <sync.h>
 
-class CNetFulfilledRequestManager;
+#include <memory>
-extern CNetFulfilledRequestManager netfulfilledman;
 
-// Fulfilled requests are used to prevent nodes from asking for the same data on sync
+template<typename T>
-// and from being banned for doing so too often.
+class CFlatDB;
-class CNetFulfilledRequestManager
+class CNetFulfilledRequestManager;
+
+class NetFulfilledRequestStore
 {
-private:
+protected:
     typedef std::map<std::string, int64_t> fulfilledreqmapentry_t;
     typedef std::map<CService, fulfilledreqmapentry_t> fulfilledreqmap_t;
 
+protected:
     //keep track of what node has/was asked for and when
     fulfilledreqmap_t mapFulfilledRequests;
     mutable RecursiveMutex cs_mapFulfilledRequests;
 
 public:
-    CNetFulfilledRequestManager() {}
+    SERIALIZE_METHODS(NetFulfilledRequestStore, obj)
-
-    SERIALIZE_METHODS(CNetFulfilledRequestManager, obj)
     {
         LOCK(obj.cs_mapFulfilledRequests);
         READWRITE(obj.mapFulfilledRequests);
     }
 
+    void Clear();
+
+    std::string ToString() const;
+};
+
+// Fulfilled requests are used to prevent nodes from asking for the same data on sync
+// and from being banned for doing so too often.
+class CNetFulfilledRequestManager : public NetFulfilledRequestStore
+{
+private:
+    using db_type = CFlatDB<NetFulfilledRequestStore>;
+
+private:
+    const std::unique_ptr<db_type> m_db;
+    const bool is_valid{false};
+
+public:
+    explicit CNetFulfilledRequestManager(bool load_cache);
+    ~CNetFulfilledRequestManager();
+
+    bool IsValid() const { return is_valid; }
+    void CheckAndRemove();
+
     void AddFulfilledRequest(const CService& addr, const std::string& strRequest);
     bool HasFulfilledRequest(const CService& addr, const std::string& strRequest);
 
     void RemoveAllFulfilledRequests(const CService& addr);
 
-    void CheckAndRemove();
-    void Clear();
-
-    std::string ToString() const;
-
     void DoMaintenance();
 };
 
+extern std::unique_ptr<CNetFulfilledRequestManager> netfulfilledman;
+
 #endif // BITCOIN_NETFULFILLEDMAN_H

src/rpc/evo.cpp

@@ -1303,7 +1303,8 @@ static UniValue BuildDMNListEntry(CWallet* pwallet, const CDeterministicMN& dmn,
     }
 #endif
 
-    auto metaInfo = mmetaman.GetMetaInfo(dmn.proTxHash);
+    CHECK_NONFATAL(mmetaman != nullptr);
+    auto metaInfo = mmetaman->GetMetaInfo(dmn.proTxHash);
     o.pushKV("metaInfo", metaInfo->ToJson());
 
     return o;

src/spork.cpp

@@ -6,6 +6,7 @@
 
 #include <chainparams.h>
 #include <consensus/params.h>
+#include <flat-database.h>
 #include <key_io.h>
 #include <logging.h>
 #include <messagesigner.h>
@@ -25,6 +26,8 @@
 
 std::unique_ptr<CSporkManager> sporkManager;
 
+const std::string SporkStore::SERIALIZATION_VERSION_STRING = "CSporkManager-Version-2";
+
 std::optional<SporkValue> CSporkManager::SporkValueIfActive(SporkId nSporkID) const
 {
     AssertLockHeld(cs);
@@ -56,7 +59,7 @@ std::optional<SporkValue> CSporkManager::SporkValueIfActive(SporkId nSporkID) co
     return std::nullopt;
 }
 
-void CSporkManager::Clear()
+void SporkStore::Clear()
 {
     LOCK(cs);
     mapSporksActive.clear();
@@ -65,10 +68,32 @@ void CSporkManager::Clear()
     // we should not alter them here.
 }
 
+CSporkManager::CSporkManager() :
+    m_db{std::make_unique<db_type>("sporks.dat", "magicSporkCache")}
+{
+}
+
+CSporkManager::~CSporkManager()
+{
+    if (!is_valid) return;
+    m_db->Store(*this);
+}
+
+bool CSporkManager::LoadCache()
+{
+    assert(m_db != nullptr);
+    is_valid = m_db->Load(*this);
+    if (is_valid) {
+        CheckAndRemove();
+    }
+    return is_valid;
+}
+
 void CSporkManager::CheckAndRemove()
 {
     LOCK(cs);
-    assert(!setSporkPubKeyIDs.empty());
+
+    if (setSporkPubKeyIDs.empty()) return;
 
     for (auto itActive = mapSporksActive.begin(); itActive != mapSporksActive.end();) {
         auto itSignerPair = itActive->second.begin();
@@ -327,7 +352,7 @@ bool CSporkManager::SetPrivKey(const std::string& strPrivKey)
     return true;
 }
 
-std::string CSporkManager::ToString() const
+std::string SporkStore::ToString() const
 {
     LOCK(cs);
     return strprintf("Sporks: %llu", mapSporksActive.size());

src/spork.h

@@ -20,6 +20,8 @@
 #include <vector>
 
 class CConnman;
+template<typename T>
+class CFlatDB;
 class CNode;
 class CDataStream;
 class PeerManager;
@@ -155,41 +157,17 @@ public:
     void Relay(CConnman& connman) const;
 };
 
-/**
+class SporkStore
- * CSporkManager is a higher-level class which manages the node's spork
- * messages, rules for which sporks should be considered active/inactive, and
- * processing for certain sporks (e.g. spork 12).
- */
-class CSporkManager
 {
-private:
+protected:
-    static constexpr std::string_view SERIALIZATION_VERSION_STRING = "CSporkManager-Version-2";
+    static const std::string SERIALIZATION_VERSION_STRING;
-
-    mutable Mutex cs_mapSporksCachedActive;
-    mutable std::unordered_map<const SporkId, bool> mapSporksCachedActive GUARDED_BY(cs_mapSporksCachedActive);
-
-    mutable Mutex cs_mapSporksCachedValues;
-    mutable std::unordered_map<SporkId, SporkValue> mapSporksCachedValues GUARDED_BY(cs_mapSporksCachedValues);
 
     mutable Mutex cs;
 
     std::unordered_map<uint256, CSporkMessage, StaticSaltedHasher> mapSporksByHash GUARDED_BY(cs);
     std::unordered_map<SporkId, std::map<CKeyID, CSporkMessage> > mapSporksActive GUARDED_BY(cs);
 
-    std::set<CKeyID> setSporkPubKeyIDs GUARDED_BY(cs);
-    int nMinSporkKeys GUARDED_BY(cs) {std::numeric_limits<int>::max()};
-    CKey sporkPrivKey GUARDED_BY(cs);
-
-    /**
-     * SporkValueIfActive is used to get the value agreed upon by the majority
-     * of signed spork messages for a given Spork ID.
-     */
-    std::optional<SporkValue> SporkValueIfActive(SporkId nSporkID) const EXCLUSIVE_LOCKS_REQUIRED(cs);
-
 public:
-
-    CSporkManager() = default;
-
     template<typename Stream>
     void Serialize(Stream &s) const LOCKS_EXCLUDED(cs)
     {
@@ -221,6 +199,50 @@ public:
      */
     void Clear() LOCKS_EXCLUDED(cs);
 
+    /**
+     * ToString returns the string representation of the SporkManager.
+     */
+    std::string ToString() const LOCKS_EXCLUDED(cs);
+};
+
+/**
+ * CSporkManager is a higher-level class which manages the node's spork
+ * messages, rules for which sporks should be considered active/inactive, and
+ * processing for certain sporks (e.g. spork 12).
+ */
+class CSporkManager : public SporkStore
+{
+private:
+    using db_type = CFlatDB<SporkStore>;
+
+private:
+    const std::unique_ptr<db_type> m_db;
+    bool is_valid{false};
+
+    mutable Mutex cs_mapSporksCachedActive;
+    mutable std::unordered_map<const SporkId, bool> mapSporksCachedActive GUARDED_BY(cs_mapSporksCachedActive);
+
+    mutable Mutex cs_mapSporksCachedValues;
+    mutable std::unordered_map<SporkId, SporkValue> mapSporksCachedValues GUARDED_BY(cs_mapSporksCachedValues);
+
+    std::set<CKeyID> setSporkPubKeyIDs GUARDED_BY(cs);
+    int nMinSporkKeys GUARDED_BY(cs) {std::numeric_limits<int>::max()};
+    CKey sporkPrivKey GUARDED_BY(cs);
+
+    /**
+     * SporkValueIfActive is used to get the value agreed upon by the majority
+     * of signed spork messages for a given Spork ID.
+     */
+    std::optional<SporkValue> SporkValueIfActive(SporkId nSporkID) const EXCLUSIVE_LOCKS_REQUIRED(cs);
+
+public:
+    CSporkManager();
+    ~CSporkManager();
+
+    bool LoadCache();
+
+    bool IsValid() const { return is_valid; }
+
     /**
      * CheckAndRemove is defined to fulfill an interface as part of the on-disk
      * cache used to cache sporks between runs. If sporks that are restored
@@ -244,7 +266,6 @@ public:
      */
     void ProcessSpork(const CNode& peer, PeerManager& peerman, CConnman& connman, CDataStream& vRecv) LOCKS_EXCLUDED(cs);
 
-
     /**
      * ProcessGetSporks is used to handle the 'getsporks' p2p message.
      *
@@ -315,11 +336,6 @@ public:
      * address in the set of valid spork signers (see SetSporkAddress).
      */
     bool SetPrivKey(const std::string& strPrivKey) LOCKS_EXCLUDED(cs);
-
-    /**
-     * ToString returns the string representation of the SporkManager.
-     */
-    std::string ToString() const LOCKS_EXCLUDED(cs);
 };
 
 #endif // BITCOIN_SPORK_H

src/test/util/setup_common.cpp

@@ -11,10 +11,13 @@
 #include <consensus/params.h>
 #include <consensus/validation.h>
 #include <crypto/sha256.h>
+#include <flat-database.h>
 #include <governance/governance.h>
 #include <index/txindex.h>
 #include <init.h>
 #include <interfaces/chain.h>
+#include <masternode/meta.h>
+#include <netfulfilledman.h>
 #include <llmq/blockprocessor.h>
 #include <llmq/chainlocks.h>
 #include <llmq/context.h>
@@ -206,6 +209,8 @@ ChainTestingSetup::ChainTestingSetup(const std::string& chainName, const std::ve
     ::sporkManager = std::make_unique<CSporkManager>();
     ::governance = std::make_unique<CGovernanceManager>();
     ::masternodeSync = std::make_unique<CMasternodeSync>(*m_node.connman, *::governance);
+    ::mmetaman = std::make_unique<CMasternodeMetaMan>(/* load_cache */ false);
+    ::netfulfilledman = std::make_unique<CNetFulfilledRequestManager>(/* load_cache */ false);
 
     m_node.creditPoolManager = std::make_unique<CCreditPoolManager>(*m_node.evodb);
 
@@ -222,6 +227,8 @@ ChainTestingSetup::~ChainTestingSetup()
     StopScriptCheckWorkerThreads();
     GetMainSignals().FlushBackgroundCallbacks();
     GetMainSignals().UnregisterBackgroundSignalScheduler();
+    ::netfulfilledman.reset();
+    ::mmetaman.reset();
     ::masternodeSync.reset();
     ::governance.reset();
     ::sporkManager.reset();