Version 2.6 is well underway. Here are some of the exciting things coming down the pipeline that you can find in beta right now!
What is Version 2.6?
The bulk of this update focuses on the backend with optimization and performance enhancements using new infrastructure and patterns. It greatly improves performance, increases responsiveness, and paves the way for a whole suite of new features that weren't feasible with existing systems. So know that while you will be seeing features mentioned in this post, they are the tip of the iceberg in terms of the improvements to Statbot that I've worked on these past few months.
As I've said in the past, version bumps can represent important milestones in Statbot's growth and development. While most of this update is invisible to the user, I can assuredly say that it is exactly that.
The website will be the first to make use of the new backend, which should be immediately apparent to all who have used the beta website in the last 24 hours.
This update contains the following parts:
- Infrastructure & Performance
- Dashboard Filters
- Analytics Page
- Statrole Compare Condition
Infrastructure & Performance
There's not much to say here since the changes will be felt rather than seen; however, I would like to speak to some of the motivation and concerns that brought this on, why I felt it was necessary to implement this now rather than later, and why you should care.
Reason 1 - Declining dashboard performance
First, there was a server that was logging a few million messages a day for some time back in November when I started this effort. It was a gigantic outlier that dwarfed everything seen before it. It was so bad that the dashboard was completely unable to load for that server and it would cause slow downs for some other users of the website. Some systems have understandably slowed down a bit over time as more and more has been demanded by them, but a dashboard being unable to load is unacceptable to me. That was the catalyst for my crusade to make it so that all servers have a nearly identical, smooth experience regardless of how much data they log. Existing methods of querying and processing data needed to be reevaluated.
Reason 2 - The new Dashboard Filters
Second, the new Dashboard Filters feature is quite demanding itself. No longer can I rely on database query caching and user patience. If someone is testing out different filters, they can't be waiting an upwards of a minute to see the results with every change. Premium settings like History and Role Filters can't worsen performance to the degree that they were. Huge servers with 500k members can't have to wait a whole minute just to see their all-time membercount. Drilldowns can't be held up because member data is being fetched painfully slowly. A system-wide overhaul and new approaches to the problems we now know about was necessary.
Reason 3 - Storage concerns
Third, the old database is currently 3.3 Terabytes. The message table alone is 1.8TB sitting at 6.3 billion rows. Honestly, I'm proud of the fact that it is as performant as it is. The decline in performance is less about the amount of data and more about the new demanding ways that the data is being used. It's both a testament to the sturdiness of existing systems and the progress Statbot has made with new features... However, with the bot in only 250k servers, this didn't bode well for growth. Doubly so when considering the upcoming Timezones integration, which will inflate the data tables. Triply so when we start talking about adding member in/out, word and character counts, and other data that would surely push that already lofty storage number even higher. A new storage solution was needed.
Reason 4 - Time for a change
Finally, I'm not the same programmer I was when I made the systems that are being overhauled/replaced. I've iterated on them to make them better over time, but sometimes a blank slate approach is necessary--such as the website update and the recent Statrole update--a full reevaluation of how things can be done, and testing out new methods of data storage and transportation in a clean environment away from old / aging patterns. It's taken some time to learn and test all the things I have, but it is well worth the effort.
And so, know that while flashy and cool features haven't been flooding out recently, your Statbot experience is about to get a major boost that will be felt, not seen.
Website
Okay, well that's a bit of a lie. It will be felt AND seen with some long-awaited features now making their way to the dashboard! Well, they've been available since a ninja release for the better part of the last 3 months, but now I'm announcing them!
Dynamic Dashboard Filters
Dashboard Filters let you change the data you're viewing on-the-fly. All the options from the Settings page are available on the dashboard; and just like Drilldowns, the Dashboard Filters are reflected in the URL so you can copy-paste it for others to see!*
To answer a question that I'm sure will be at the front of many minds, "Are there any premium restrictions"? The answer is that there are no restrictions outside of those that already exist. That is, History for data older than 30 days and Data+ for Role Filters.
* Dashboard Filters have the same permissions restrictions as Drilldowns, meaning that the member must have
Administrator, or have a role in the Analytics Roles Statbot setting, and at least one of View Server Insights
or Manage Server. Any other member will only see what the server's settings are.
Analytics Page (Drilldowns)
The new features don't stop at just Dashboard Filters! Another one of our most requested features is the ability to easily see and query all members and channels in a server. While you can now sort of do that on the regular dashboard with Dashboard Filters, there's a new Analytics page to streamline the process and surface data in a much easier way to digest.
On the Analytics page, you can see both message and voice data on the same page, and turn the various knobs and dials to get the information you need before pivoting to the Drilldowns.
This comes with some additional metrics. Data that's now available on all the Message, Voice, and Analytics pages is Unique members and Unique channels. Unique members is the number of members that participated in a given data point, and Unique channels is the same for channels.
Along with this, a powerful addition to the Analytics page is the "Not in server" value that shows you messages and voice activity that came from members that are no longer in the server. These are the first of many metrics that will add much-needed context to the plentitude of data Statbot already makes available!
Where did Engagement Go?
The new Engagement metric made available for a short time on the Analytics page is unfortunately too resource hungry to be included with the current services. The way those metrics were computed goes against the the goal of speeding up the website's performance. Because of this, it has been temporarily disabled.
That said, it's still something I very much want to do. The response for the few who used it was positive and I think that it's an innovative way to simplify the sometimes overwhelming amount of information available through Statbot. Do expect it to make its return in the future!
New Graphs
As seen on the Message, Voice, and Analytics pages, I've created a new graph component to make viewing and selecting data easier. There are also new averages on the charts:
- An overall Average for the data in view; and
- the ability to add you own custom Moving Averages
Moving averages help to smooth out otherwise chaotic data and gives you an idea on if values are trending up or down. Setting your own values let's you tailor the length of trend you want to see. Moving averages are also save to the URL to share with others.
Statrole Compare Condition (Statroles+)
Another ninja release currently in beta was that of the Statrole Compare condition. This is a very special condition that allows you to mix and match different stat types (Message, Voice, Activity), weight them, and then compare them to another set of stats in order to determine who gets the role. This makes it so that you can do things like give roles based on which channels people spend most of their time, rather than just a simple white / blacklist.
No other condition lets you combine the values of stats, so this makes for new and exciting possibilities to leverage member activity in role-giving systems.
A Secret??
There's a secret change that will be added soon!
Bot?
The bot currently does not use the new infrastructure so unfortunately there isn't any performance benefits there, yet.
What's Next?
The release of version 2.6 marks the end of the current work on the website and a move to giving the bot the TLC it much deserves. This includes adding tracking to support timezones, updating commands with new graphs and visuals, adding slash commands and component support, bringing localization support to Statbot for the first time, and much, much more!
This work will rival the work I did rebuilding the website from the ground up last year. Look forward to the release of version 2.7 coming eventually™ to a Statbot near you!
Keep up-to-date on Statbot changes by subscribing to the @Bot Updates role in the Statbot support server!