Shub's logo

Up And Up up-n-up

09 May, 2020

4 min read

This blog is a Work in Progress, and I am hoping to work on it part by part when I will really start implementing th idea mentioned in here. Gradually noting down what hurdles I faced or if some changes I made, that needs to be reflected here as well.

⚠️  For now, read it at your own risk!!!

Up and Up our logs you go.
To give an insight
Of what should you Hope. 🎼

What/Why the hell???

This project is meant to get essential logs from client's browser, so as to later verify what went wrong. Usually, in Front-end development, developers tend to disable logs to improve performance and many other good reasons, but in initial days of development of a product sometimes requires us to monitor how tha app is performing on the end-user's browsers. This project is developed to support an idea that I had yesterday (it's similar to what Google Analytics or Rollbar or Sentry does), to get small chunks of performance related data, on a constant interval, that we can later on verify what went wrong, or where an improvement can be made.

Pseudo Implementation:

Following implementation is not tested or written in real code. It's just for explanatory purpose.
Payload structures:
DefaultPayload

This payload will be extended by every other LogPayload that end-user would want to log to the service.

interface DefaultPayload {
  /**
   * Name of the OS, like Windows/Android etc.
   */
  client_os: string;
  /**
   * Version of the OS being used, would come handy to figure out what CPU processor might
   * be used
   */
  client_version: number | null;
  /**
   * Browser details, still don't know how to read this
   */
  user_agent: string;
  /**
   * Following Log Level doesn't actually refer to Log Levels, but they denote the weight
   * of a Log, depending on the feature.
   *
   * `debug`: being the least weighted, and these logs should be removed, once we are done with improvements
   * `error`: being the medium weighted.
   * `perf`: being the medium weighted, which we can keep to track performance details of end-user's per browser
   * `data`: Never to be removed, they denote a Log, required by Data-Analysts.
   */
  log_level: 'error' | 'debug' | 'perf' | 'data';
}
LogPayload

This is a custom payload and might differ as per developer's implementation, since various apps have various logs they want to track or later on do elastic search or something.

interface LogPayload extends DefaultPayload {
  /**
   * Could relate to various features/modules being used in an App.
   */
  feature: string;
  /**
   * Internal task related to a feature like `chat-send` or `blog-post`
   * This is an array, because in a feature we can have multiple sub-features batched into one
   * single API call.
   *
   * Having an array of sub-features will also help us to batch performance logs into one single API call
   *
   * sub_features are basically a one-to-one map for `measures`, but it won't be case everytime.
   */
  sub_features?: string[];
  /**
   * User's ID to log for
   */
  user_id: number;
  /**
   * Current User's Email
   */
  email: string;
  /**
   * Performance measures: providing us Time Deltas, so as to figure
   * out if any feature or sub-feature is taking time, and what exactly is causing it
   *
   * Measure data is not required for log_level: ['debug', 'data'];
   */
  measures?: MeasurePayload[];
  /**
   * Extra meta data, could be present or not, nobody knows ;P
   */
  meta?: Record<string, any>;
};
MeasurePayload

This payload defines the details on what performance measure were for a particular sub-feature. Will help to get details on performance of our app.

interface MeasurePayload {
  /**
   * name: correlates to the sub_feature we are working on, so that it is easier to understand
   * for which feature time delta was taken for.
   */
  name: string;
  /**
   * Time Delta, between Start Time and End Time for this sub-feature.
   *
   * delta = 0, if sub_feature is instantaneous, i.e., we don't want to log perf delta
   */
  delta: number;
  /**
   * Should always be an timestamp number
   *
   * whether or not delta is present, we should have Timestamp details when logging any
   * sub-feature, for later references
   *
   * Maps to `Date.note()`
   */
  timestamp: string;
};

We need to create server, which will listen to some HTTP endpoints

Endpoints like:

Following APIs will need an array as payload, as they will have a batch of data to be sent to server.

Whatever Logs we receive in server, can be put into a log file, that can be dumped into some warehouse or can be processed in an Elastic Search or completely sent to some Analytics server like Google Analytics without saving them to file.

What I am thinking is, any payload that come to /data or /bulk/data will be sent to whatever analytics backend we will link to our server, like Google Analytics or Segment or anything.


We need to create a Frontend library as well that will manage all these calls to backend

Since we want to send single or batched payloads to server, we need a library to manage that burden from a dev, and provide a clean API that the dev can usee to

Concept

The thought behind building this library is simple:

Flow Diagram for Log Upload

Flow Diagram for Log Upload

If we do use Webworkers, someday in future, all the above logic will become parallel and it will become trivial to use Mutex/Locks to manage manipulation of queues, which also gives us more control over when the data chunk should be extracted, i.e. should we get the chunk after 2 seconds or should we get the chunk when it crosses a size threshold of let's say 100kb, so that the API calls we make are more effective.
Solution

Using RxJS, to simplify the concept of buffering. Understanding RxJS took me a lot of time. There are four main concepts for RxJS.

  1. Observable/Observer (Publisher/Subscriber)
  2. Cold and Hot Publisher
  3. Schedulers
  4. Operators

I will discuss these in a separate Note. For now we will just talk about buffering.

© Copyright 2020 Subroto Biswas

Share