The right way to load and use "pickle" files

Comments

• ptunney Posts: 246

  July 2019

  Due to security restrictions, there is no way to upload anything to the site that is a binary file.
  Internal users, like Trevor do not have those restrictions so I can appreciate the frustration.
  But if you are using our system for machine learning and creating a pickle and you are stuck at the point of uploading a pickle, get in touch with us at customer_success@cloudquant.com, we are very interested in what people are doing with our system and we want to help you achieve your goals.
• an623267 Posts: 7

  July 2019

  @pt120221 said:
  Due to security restrictions, there is no way to upload anything to the site that is a binary file.
  Internal users, like Trevor do not have those restrictions so I can appreciate the frustration.
  But if you are using our system for machine learning and creating a pickle and you are stuck at the point of uploading a pickle, get in touch with us at customer_success@cloudquant.com, we are very interested in what people are doing with our system and we want to help you achieve your goals.

  Thank you. For now I managed to use the model by extracting weights from it and saving them in CSV. And then making predictions using numpy matrix multiplication in your enviroment. It's ok for the simple model that I'm trying to use right now.
• an623267 Posts: 7

  July 2019 edited July 2019

  Maybe someone will find this useful. Works only for simple dense model.

  I just save weights by Keras Model's method in .h5 format:

  model.save_weights('model_weights.h5')
  

  Then extract the parameters from file:

  import h5py
  parameters = {}
  
  with h5py.File('model_weights.h5') as f:
      count = 0
      for layer, group in f.items():
          if not len(group.keys()) > 0:
                  continue
          count += 1
          for params_name in group.keys():
                  params = group[params_name]
                  for p_name in params.keys():
                      if "kernel" in p_name:
                          parameters["W" + str(count)] = params.get(p_name)[:].T
                      if "bias" in p_name:
                          parameters["b" + str(count)] = params.get(p_name)[:].reshape(-1,1)
  

  Save it to CSV:

  import pandas as pd
  for key, item in parameters.items():
      df = pd.DataFrame(item)
      df.to_csv(path_or_buf= "name_" + key + ".csv", header=True, index=False)
  

  In the CloudQuant enviroment:

          import numpy as np
          @classmethod
          def on_strategy_start(cls, md, service, account):
              cls.parameters = {}
  
              cls.parameters["W1"] = cls.get_parameters(service.read_file("name_W1.csv", format='csv'))
              cls.parameters["b1"] = cls.get_parameters(service.read_file("name_b1.csv", format='csv'))
              cls.parameters["W2"] = cls.get_parameters(service.read_file("name_W2.csv", format='csv'))
              cls.parameters["b2"] = cls.get_parameters(service.read_file("name_b2.csv", format='csv'))
              cls.parameters["W3"] = cls.get_parameters(service.read_file("name_W3.csv", format='csv'))
              cls.parameters["W4"] = cls.get_parameters(service.read_file("name_W4.csv", format='csv'))
              cls.parameters["b4"] = cls.get_parameters(service.read_file("name_b4.csv", format='csv'))
              cls.parameters["W5"] = cls.get_parameters(service.read_file("name_W5.csv", format='csv'))
              cls.parameters["b5"] = cls.get_parameters(service.read_file("name_b5.csv", format='csv'))
  
          @staticmethod
          def get_parameters(file):
              parameters = None
              for row in file:
                  list = []
                  for i in range(len(row)):
                      list.append(float(row[str(i)]))
                  parameters = np.array(list) if parameters is None else np.vstack([parameters, list]) 
              return parameters
  
          @classmethod
          def nn_forward(cls, X, parameters):
              L = len(parameters)//2
              A_prev = X
              for i in range(1,L):
                  Z = np.dot(parameters["W" + str(i)], A_prev) + parameters["b" + str(i)]
                  A_prev = cls.relu(Z)
  
              ZL = np.dot(parameters["W" + str(L)], A_prev) + parameters["b" + str(L)]
              AL = cls.softmax(ZL)
  
              return AL
  
          @staticmethod
          def relu(X):
              return X*(X > 0)
  
          @staticmethod
          def softmax(X):
              return np.exp(X)/(np.exp(X).sum(axis=0))
  
          def on_minute_bar(self, event, md, order, service, account, bar):
              cls = self.__class__
              prediction = cls.nn_forward(X, cls.parameters)
  

  X shape: (n_features,1)