Planet Gamedev

Geeks3D Forums

(Shadertoy) Round-up May 2015

May 24, 2015 05:03 PM

My personal favorites this month.
All running on Nexus 4 , AMD Zacate APU and NVIDIA Maxwell

PBR_Test
Hyper dodecahedron
[url=https://www.shadertoy.com/vi...

(WebGL) Raptor running in WebGL using Creature Animation Tool

May 24, 2015 04:45 PM

Raptor running in WebGL using Creature Animation Tool

[via]

[img]https://c1.stat...

(WebGL) Heat snakes - fun with diffusion shaders

May 24, 2015 03:31 PM

Heat snakes - fun with diffusion shaders!

"A WebGL diffusion simulation with added visual touches."

[...

David Perry Blog

Visual Recognition

by David Perry at May 23, 2015 05:36 PM

Was just sent this by a friend, it’s a great test of visual recognition at the moment!

Camfind managed to recognize Crocoduck!

It’s fun to see the progress on this sector of research as it’s going to fundamentally change things when computers can really see and understand what they see.  Now driverless cars can avoid crocoducks!

http://venturebeat.com/2015/05/16/how-stephen-wolframs-image-recognition-tool-performs-against-5-alternatives/


Filed under: ARTICLES/LINKS Tagged: computer vision, visual recognition

Geeks3D Forums

Intel Test Driver for HDMI Display Detection Issues

May 23, 2015 07:16 AM

Test Driver for HDMI Display Detection Issues

In an ongoing effort to resolve HDMI display detection some users are experiencing, we are releasing a set of test drivers for Sandybridge, Ivybridge, H...

NVIDIA GeForce Hotfix driver 353.00

May 23, 2015 04:59 AM

Quote
GeForce Hotfix driver 353.00

This is a GeForce Hot Fix driver, version 353.00 that addresses the following issues:[/...

Gamasutra Feature Articles

Don't Miss: What does a game designer do, exactly?

May 22, 2015 08:22 PM

In this timeless blog, designer Liz England takes a run at breaking down the many roles and responsibilities a designer can have in an effort to answer a seemingly ever-present question. ...

Get a job: Be a Cinematic Artist for Telltale Games

May 22, 2015 07:28 PM

The studio responsible for (among other things) Game of Thrones and The Walking Dead are aiming to hire a cinematic artist to join the team at Telltale's San Rafael, CA office. There will be a test. ...

Video: Designing an immersive user interface for Hearthstone

May 22, 2015 07:18 PM

Hearthstone senior UI designer Derek Sakamoto breaks down Blizzard's approach to UI design at GDC 2015, in the process shedding light on how you can best craft a great UI for your own game. ...

Postmortem: Pac-Man, Iwatani's rhapsody in yellow

May 22, 2015 06:29 PM

35 years ago today, Namco released Pac-Man in Japan and jumpstarted a new age of game development. In this classic 2005 Game Developer Magazine feature, Pac-Man creator Toru Iwatani explains how. ...

Game Design Aspect of the Month

Leading by Emotion

by noreply@blogger.com (Sande Chen) at May 22, 2015 05:54 PM

In this article, game designer Sande Chen wonders about the role of emotions in designing a game.

One particular way of teaching creative writing is to ask students to explore the essence of emotion and thus, I attended a Meditation and Writing Retreat last December to reach back into my memory and recall the sensations associated with the emotion at the time.  How did this emotion manifest in the body?  Did it cause dry mouth, sniffling, a tightness in the chest, or blank eyes?  Writing about emotion was an instant connection to my creative fury. 

While emotions are an accepted starting point for creative writing, I realize emotions aren't the genesis for most games.  A tech demo is an accepted starting point.  In fact, most genres of games are defined by gameplay rather than the feelings elicited by the game.  The only exception might be horror games, which follow traditional genre fiction categorization.  It's for this very reason that I organized a panel to explore the nature of horror games.  (Stay tuned for more info!)  

But what if emotions played a more important role in game creation?  After all, according to American author James Gunn, people look to fiction to engage in an emotional experience.  Why shouldn't it be the same for games?  And I'm not strictly talking about "making the player cry," but simply about connecting on an emotional level. As game designer Reid Kimball says in Breaking the Vicious Cycle, let's inspire players and go beyond grinding.
 
For many people, Jenova Chen's games are inspirational.  In preparation for my article, Towards More Meaningful Games: A Multidisciplinary Approach, I talked to Jenova Chen about preproduction.  He showed me this Emotional Intensity Graph he made during the conception phase.  What's interesting is that this isn't a map of a character's emotions, but of the player's intended emotions.


Filmmakers and authors need to carefully craft an audience's emotional expectations.  I assume that game designers should do the same.  Should we expect anything less?

Sande Chen is a writer and game designer whose work has spanned 10 years in the industry. Her credits include 1999 IGF winner Terminus, 2007 PC RPG of the Year The Witcher, and Wizard 101. She is one of the founding members of the IGDA Game Design SIG.

Gamasutra Feature Articles

Laid-off Zynga devs get a fresh start at a fantasy sports company

May 22, 2015 05:42 PM

A former Zynga employee tells Re/code that roughly 38 of the 42 devs who lost their jobs when Zynga shuttered its Orlando studio this month have gone on to join fantasy sports game company Fanduel. ...



Who am I to tell their story? Writing for diversity

May 22, 2015 05:40 PM

"I will specifically highlight my insecurities while writing about people of color who lived in the age of colonialism. An honest and realistic answer to the question: Who am I to tell their story?" ...

Opinion: Why Guild Wars 2's 'execution'-style ban was a bad idea

May 22, 2015 05:23 PM

ArenaNet recently banned an infamous hacker from its MMO Guild Wars 2 -- by way of virtual execution. But public humiliation is not really the way to go about it, argues Katherine Cross. ...

Learning a lot, fast by jamming in VR

May 22, 2015 04:57 PM

"I cannot remember when in the past I learned so much in just one month. We proved we are capable of doing a 3D and VR game in this young technology." ...

Geeks3D Forums

(Demoscene) Birdie, Outline, qbparty 2015

May 22, 2015 04:09 PM

Birdie 2015

Outline 2015

qbparty 2015


I would have tested more Jav...

Timothy Lottes

The Other Project : Cleaned Up

by Timothy Lottes (noreply@blogger.com) at May 22, 2015 04:49 PM

From the CLP Motorsports open house,




Gamasutra Feature Articles

Tackling mental illness in game design: A Vanishing Point postmortem

May 22, 2015 03:42 PM

A tough task: Representing mental illness with respect and relevance. "Vanishing Point lacked the laser-precise direction to unite its constituent parts into something thematic, relevant, and positive." ...



Point-and-click in the age of touch: Interface design for adventure games

May 22, 2015 02:07 PM

"Create something new nobody has seen before that ultimately fits your game! Seriously! I'd like to encourage that! There should be more adventure games with innovative and new interfaces." ...

Game From Scratch

Godot Engine release version 1.1

by Mike@gamefromscratch.com at May 22, 2015 12:47 PM

 

Following the Beta release a few weeks back, the Godot Engine team have just released Godot Engine 1.1.

 

The highlights of the release are:

  • Rewritten Auto-Completion in the Code-Editor. Supports a lot of scenarios and perform smart-completion of node types if a scene where the script is being used is open.
  • Visual Shader Editor (Edit shaders connecting nodes)
  • New API in OS for managing the screens and window, with multi-monitor support.
  • Largely rewritten 2D engine, with support for:
    • Shaders (Visual and Code)
    • 2D Materials
    • 2D Independent Z ordering per-node.
    • 2D Lights
    • 2D Shadows with Polygonal Occluders
    • 2D Normal Mapping
    • Back-Buffer compositing for shaders that read from screen (allows all sorts of post-processing effects).
    • Improved Isometric TileMap Support (proper Z ordering of tiles and children nodes).
    • Distance-Field font support.
  • New 2D Navigation Polygon support, for efficient 2D pathfinding. Navigation Polygons can be edited visually and combined, disabled, etc.
  • Improved Usability in 2D Physics API:
    • Area2D and RigidBody2D can receive input events
    • Area2D can detect overlap with another Area2D
  • New Dark Theme
  • Much Improved Blender Collada Exporter (BetterCollada).
  • Large amount of bug fixes and smaller improvements.

 

Full (enormous) changelog since 1.0 here.

 

They also put together a video showcasing the new features:

 

Of course, if you are interested in learning more GFS has an extensive Godot Tutorial Series to get you started.

Gamasutra Feature Articles

Starting up a game business: Contemplating processes

May 22, 2015 11:01 AM

A blog about "the dangers of processes, what is important to consider about them and how to avoid pitfalls when using any given methodology." ...



cbloom rants

05-21-15 - Umm

by cbloom (noreply@blogger.com) at May 22, 2015 11:39 AM

I sent a lawyer an email yesterday.

Today they sent me back an email saying :

"I need your email address so I can send you the documents you need to sign"

Umm... you are not inspiring great confidence in your abilities.

Also, pursuant to my last post about spam - pretty much all my correspondence with lawyers over the past few months, Google decides to put in the spam folder. I keep thinking "WTF why didn't this lawyer get back to me - oh crap, go check the spam". Now, I'm totally down with the comic social commentary that Google is making ("ha ha, all email from lawyers is spam, amirite? lol"). But WTF your algorithms are insanely broken. I mean, fucking seriously you suck so bad.

05-21-15 - Software Patents are Fucking Awesome

by cbloom (noreply@blogger.com) at May 22, 2015 11:39 AM

Awesome. Spotted on encode.ru. It was inevitable I suppose :

"System and method for compressing data using asymmetric numeral systems with probability distributions"

By these tards :

Storleap

Someone in the UK go over and punch them in the balls.

For those not aware of the background, ANS is probably the biggest invention in data compression in the last 20 years. Its inventor (Jarek Duda) has explicitly tried to publish it openly and make it patent-free, because he's awesome.

In the next 10 years I'm sure we will get patents for "using ANS with string-matching data compression", "using ANS with block mocomp data compression", "using ANS as a replacement for Huffman coding", "deferred summation with ANS", etc. etc. Lots of brilliant inventions like that. Really stimulating for innovation.

(as has happened over and over in data compression, and software in general in the past; hey let's take two obvious previously existing things; LZ string matching + Huffman = patent. LZ + hash table = patent. JPEG + arithmetic = patent. Mocomp + Huffman = patent. etc. etc.)

(often glossed over in the famous Stac-Microsoft suit story is the question of WHAT THE FUCK the LZS patent was supposed to be for? What was the invention there exactly? Doing LZ with a certain fixed bit encoding? Umm, yeah, like everyone does?)

Our patent system is working great. It obviously protects and motivates the real inventors, and doesn't just act as a way for the richest companies to lock in semi-monopolies of technologies they didn't even invent. Nope.

Recently at RAD we've made a few innovations related to ANS that are mostly in the vein of small improvements or clever usages, things that I wouldn't even imagine to patent, but of course that's wrong.

I've also noticed in general a lot of these vaporware companies in the UK. We saw one at RAD a few years ago that claimed to use "multi-dimensional curve interpolation for data compression" or some crackpot nonsense. There was another one that used alternate numeral systems (not ANS, but p-adic or some such) for god knows what.

They generally get some PhD's from Cambridge or whatever to be founders. They bring a bunch of "industry luminaries" on the board. They patent a bunch of nonsense. And then ...

... profit? There's a step missing where they actually ever make anything that works. But I guess sometimes they get bought for their vapor, or they manage to get a bullshit patent that's overly-general on something they didn't actually invent, and then they're golden.

I wonder if these places are getting college-backed "incubation" incentives? Pretty fucking gross up and down and all around. Everyone involved is scum.


On a more practical note, if anyone knows the process for objecting to a patent in the UK, chime in.

Also, shame on us all for not doing more to fight the system. All our work should be going in the Anti-Patent Patent Pool.

05-21-15 - LZ-Sub

by cbloom (noreply@blogger.com) at May 22, 2015 11:39 AM

LZ-Sub decoder :


delta_literal = get_sub_literal();

if ( delta_literal != 0 )
{
    *ptr++ = delta_literal + ptr[-lastOffset];
}
else // delta_literal == 0
{
    if ( ! get_offset_flag() )
    {
        *ptr++ = ptr[-lastOffset];
    }
    else if ( get_lastoffset_flag() )
    {
        int lo_index = get_lo_index();
        lastOffset = last_offsets[lo_index];
        // do MTF or whatever using lo_index
        
        *ptr++ = ptr[-lastOffset];
        // extra 0 delta literal implied :
        *ptr++ = ptr[-lastOffset];
    }
    else
    {
        lastOffset = get_offset();
        // put offset in last_offsets set
        
        *ptr++ = ptr[-lastOffset];
        *ptr++ = ptr[-lastOffset];
        // some automatic zero deltas follow for larger offsets
        if ( lastOffset > 128 )
        {
            *ptr++ = ptr[-lastOffset];
            if ( lastOffset > 16384 )
            {
                *ptr++ = ptr[-lastOffset];
            }
        }   
    }

    // each single zero is followed by a zero runlen
    //  (this is just a speed optimization)
    int zrl = get_zero_runlen();
    while(zrl--)
        *ptr++ = ptr[-lastOffset];
}

This is basically LZMA. (sub literals instead of bitwise-LAM, but structurally the same) (also I've reversed the implied structure here; zero delta -> offset flag here, whereas in normal LZ you do offset flag -> zero delta)

This is what a modern LZ is. You're sending deltas from the prediction. The prediction is the source of the match. In the "match" range, the delta is zero.

The thing about modern LZ's (LZMA, etc.) is that the literals-after-match (LAMs) are very important too. These are the deltas after the zero run range. You can't really think of the match as just applying to the zero-run range. It applies until you send the next offset.

You can also of course do a simpler & more general variant :

Generalized-LZ-Sub decoder :


if ( get_offset_flag() )
{
    // also lastoffset LRU and so on not shown here
    lastOffset = get_offset();
}

delta_literal = get_sub_literal();

*ptr++ = delta_literal + ptr[-lastOffset];

Generalized-LZ-Sub just sends deltas from prediction. Matches are a bunch of zeros. I've removed the acceleration of sending zero's as a runlen for simplicity, but you could still do that.

The main difference is that you can send offsets anywhere, not just at certain spots where there are a bunch of zero deltas generated (aka "min match lengths").

This could be useful. For example when coding images/video/sound , there is often not an exact match that gives you a bunch of exact zero deltas, but there might be a very good match that gives you a bunch of small deltas. It would be worth sending that offset to get the small deltas, but normal LZ can't do it.

Generalized-LZ-Sub could also give you literal-before-match. That is, instead of sending the offset at the run of zero deltas, you could send it slightly *before* that, where the deltas are not zero but are small.

(when compressing text, "sub" should be replaced with some kind of smart lexicographical distance; for each character precompute a list of its most likely substitution character in order of probability.)

LZ is a bit like a BWT, but instead of the contexts being inferred by the prefix sort, you transmit them explicitly. Weird.

05-17-15 - The Google Spam Filter is Intentionally Bad

by cbloom (noreply@blogger.com) at May 22, 2015 11:39 AM

I'm convinced at this point that Google intentionally filters spam wrong.

Not in a nefarious way, like haha we're going to send your good mails to "spam" and let the crap through! Take that!

But actually in a sort of more deeply evil way. A capitalist way. They specifically *want* to allow through mass-mailings from corporations that are they do not consider spam.

In my opinion, those are all spam. There is not a single corporate mass-mailing that I ever intentionally subscribed to.

Basically there's a very very easy spam filtering problem :

Easy 1. Reject all mass-mailings. Reject all mailings about sales, products, offers. Reject all mailings about porn or penises or nigerian princes.

Easy 2. Allow through all mail that's hand-written by a human to me. Particularly to one that I have written to in the past.

That would be fine with me. That would get 99.99% of it right for me.

They don't want to solve that problem. Instead they try to solve the much-harder problem of allowing through viagra offers that are for some reason not spam. For the email user who *wants* to get mass-mail offers of 50% off your next order.

I just don't understand how "yeah, let's go out to dinner" from my friend, who is responding with quote to a fucking mail that I sent goes in the in the Spam box, but "get direct email mass-marketing secrets to double your business!" goes in my inbox. How can it be so bad, I just really don't understand it. Fucking the most basic keyword include/exclude type of filter could do better.

I should have just written my own, because it's the kind of problem that you want to be constantly tweaking on. Every time a mail is misclassified, I want to run it through my system and see why that happened and then try to fix it.

It would be SOOO fucking easy for them. Being in a position as a central mail processor, they can tell which mails are unique and which are mass-sent, and just FUCKING BLOCK ALL THE MASS-SENT MAIL. God dammit. You are fucking me up and I know you're doing it intentionally. I hate you.


I mean, fuck. It's ridiculous.

They are responding to a mail I sent. The mail I sent is fucking quoted right there. I sent the fucking mail from gmail so you can confirm it's for real. I sent to their address with gmail. AND YOU PUT THEIR REPLY IN SPAM. WTF WTF WTF

But this is not spam :

Report: creative teamwork is easier with cloud-based apps

Businesses Increase Screening of Facebook, Twitter Before Hiring

Trying to solve the Prospecting Paradox?

I'd like to add you to my professional network on LinkedIn


Maybe I'm being a bit overly simplistic and harsh. Maybe there are mass-mailings that look spammish, but you actually want to get? Like, your credit card bill is due?

I'm not sure. I'm not sure that I ever need to get any of that. I don't need those "shipping confirmation" emails from Amazon. If they just all got filed to the "mass mail" folder, I could go look for them when I need them.


I want to make my own private internet. And then not allow anyone else to use it because you'd all just fuck it up.

05-16-15 - Threading Primitive - monitored semaphore

by cbloom (noreply@blogger.com) at May 22, 2015 11:39 AM

A monitored semaphore allows two-sided waiting :

The consumer side decs the semaphore, and waits on the count being positive.

The producer side incs the semaphore, and can wait on the count being a certain negative value (some number of waiting consuming).

Monitored semaphore solves a specific common problem :

In a worker thread system, you may need to wait on all work being done. This is hard to do in a race-free way using normal primitives. Typical ad-hoc solutions may miss work that is pushed during the wait-for-all-done phase. This is hard to enforce, ugly, and makes bugs. (it's particularly bad when work items may spawn new work items).

I've heard of many ad-hoc hacky ways of dealing with this. There's no need to muck around with that, because there's a simple and efficient way to just get it right.

The monitored semaphore also provides a race-free way to snapshot the state of the work system - how many work items are available, how many workers are sleeping. This allows you to wait on the joint condition - all workers are sleeping AND there is no work available. Any check of those two using separate primitives is likely a race.

The implementation is similar to the fastsemaphore I posted before.

"fastsemaphore" wraps some kind of underlying semaphore which actually provides the OS waits. The underlying semaphore is only used when the count goes negative. When count is positive, pops are done with simple atomic ops to avoid OS calls. eg. we only do an OS call when there's a possibility it will put our thread to sleep or wake a thread.

"fastsemaphore_monitored" uses the same kind atomic variable wrapping an underlying semaphore, but adds an eventcount for the waiter side to be triggered when enough workers are waiting. (see who ordered event count? )

Usage is like this :


To push a work item :

push item on your queue (MPMC FIFO or whatever)
fastsemaphore_monitored.post();

To pop a work item :

fastsemaphore_monitored.wait();
pop item from queue

To flush all work :

fastsemaphore_monitored.wait_for_waiters(num_worker_threads);

NOTE : in my implementation, post & wait can be called from any thread, but wait_for_waiters must be called from only one thread. This assumes you either have a "main thread" that does that wait, or that you wrap that call with a mutex.

template typename t_base_sem>
class fastsemaphore_monitored
{
    atomicS32> m_state;
    eventcount m_waiters_ec;
    t_base_sem m_sem;

    enum { FSM_COUNT_SHIFT = 8 };
    enum { FSM_COUNT_MASK = 0xFFFFFF00UL };
    enum { FSM_COUNT_MAX = ((U32)FSM_COUNT_MASK>>FSM_COUNT_SHIFT) };
    enum { FSM_WAIT_FOR_SHIFT = 0 };
    enum { FSM_WAIT_FOR_MASK = 0xFF };
    enum { FSM_WAIT_FOR_MAX = (FSM_WAIT_FOR_MASK>>FSM_WAIT_FOR_SHIFT) };

public:
    fastsemaphore_monitored(S32 count = 0)
    :   m_state(countFSM_COUNT_SHIFT)
    {
        RL_ASSERT(count >= 0);
    }

    ~fastsemaphore_monitored()
    {
    }

public:

    inline S32 state_fetch_add_count(S32 inc)
    {
        S32 prev = m_state($).fetch_add(incFSM_COUNT_SHIFT,mo_acq_rel);
        S32 count = ( prev >> FSM_COUNT_SHIFT );
        RR_ASSERT( count  0 || ( (U32)count  (FSM_COUNT_MAX-2) ) );
        return count;
    }

    // warning : wait_for_waiters can only be called from one thread!
    void wait_for_waiters(S32 wait_for_count)
    {
        RL_ASSERT( wait_for_count > 0 && wait_for_count  FSM_WAIT_FOR_MAX );
        
        S32 state = m_state($).load(mo_acquire);
        
        for(;;)
        {
            S32 cur_count = state >> FSM_COUNT_SHIFT;

            if ( (-cur_count) == wait_for_count )
                break; // got it
        
            S32 new_state = (cur_countFSM_COUNT_SHIFT) | (wait_for_count  FSM_WAIT_FOR_SHIFT);
            
            S32 ec = m_waiters_ec.prepare_wait();
            
            // double check and signal what we're waiting for :
            if ( ! m_state.compare_exchange_strong(state,new_state,mo_acq_rel) )
                continue; // retry ; state was reloaded
            
            m_waiters_ec.wait(ec);
            
            state = m_state($).load(mo_acquire);
        }
        
        // now turn off the mask :
        
        for(;;)
        {
            S32 new_state = state & FSM_COUNT_MASK;
            if ( state == new_state ) return;
        
            if ( m_state.compare_exchange_strong(state,new_state,mo_acq_rel) )
                return; 
                
            // retry ; state was reloaded
        }
    }

    void post()
    {
        if ( state_fetch_add_count(1)  0 )
        {
            m_sem.post();
        }
    }

    void wait_no_spin()
    {
        S32 prev_state = m_state($).fetch_add((-1)FSM_COUNT_SHIFT,mo_acq_rel);
        S32 prev_count = prev_state>>FSM_COUNT_SHIFT;
        if ( prev_count = 0 )
        {
            S32 waiters = (-prev_count) + 1;
            RR_ASSERT( waiters >= 1 );
            S32 wait_for = prev_state & FSM_WAIT_FOR_MASK;
            if ( waiters == wait_for )
            {
                RR_ASSERT( wait_for >= 1 );
                m_waiters_ec.notify_all();
            }
            
            m_sem.wait();
        }
    }
    
    void post(S32 n)
    {
        RR_ASSERT( n > 0 );
        for(S32 i=0;in;i++)
            post();
    }
       
    bool try_wait()
    {
        // see if we can dec count before preparing the wait
        S32 state = m_state($).load(mo_acquire);
        for(;;)
        {
            if ( state  (1FSM_COUNT_SHIFT) ) return false;
            // dec count and leave the rest the same :
            //S32 new_state = ((c-1)FSM_COUNT_SHIFT) | (state & FSM_WAIT_FOR_MASK);
            S32 new_state = state - (1FSM_COUNT_SHIFT);
            RR_ASSERT( (new_state>>FSM_COUNT_SHIFT) >= 0 );
            if ( m_state($).compare_exchange_strong(state,new_state,mo_acq_rel) )
                return true;
            // state was reloaded
            // loop
            // backoff here optional
        }
    }
     
       
    S32 try_wait_all()
    {
        // see if we can dec count before preparing the wait
        S32 state = m_state($).load(mo_acquire);
        for(;;)
        {
            S32 count = state >> FSM_COUNT_SHIFT;
            if ( count = 0 ) return 0;
            // swap count to zero and leave the rest the same :
            S32 new_state = state & FSM_WAIT_FOR_MASK;
            if ( m_state($).compare_exchange_strong(state,new_state,mo_acq_rel) )
                return count;
            // state was reloaded
            // loop
            // backoff here optional
        }
    }
           
    void wait()
    {
        int spin_count = rrGetSpinCount();
        while(spin_count--)
        {
            if ( try_wait() ) 
                return;
        }
        
        wait_no_spin();
    }

};

05-16-15 - LZ literals after match

by cbloom (noreply@blogger.com) at May 22, 2015 11:39 AM

Some vague rambling about LAMs.

LAMs are weird.

LAM0 , the first literal after a match, has the strong exclusion property (assuming maximum match lengths). LAM0 is strictly != lolit. (lolit = literal at last offset).

LAM1, the next literal after end of match, has the exact opposite - VERY strong prediction of LAM1 == lolit. This prediction continues but weakens as you go to LAM2, LAM3, etc.

In Oodle LZNA (and in many other coders), I send a flag for (LAM == lolit) as a separate event. That means in the actual literal coding path you still have LAM1 != lolit. (the LAM == lolit flag should be context-coded using the distance from the end of the match).

In all cases, even though you know LAM != lolit, lolit is still a very strong predictor for LAM. Most likely LAM is *similar* to lolit.

LAM is both an exclude AND a predictor!

What similar means depends on the file type. In text it means something like vowels stay vowels, punctuation stays punctuation. lolit -> LAM is sort of like substituting one character change. In binary, it often means that they are numerically close. This means that the delta |LAM - lolit| is never zero, but is often small.

One of the interesting things about the delta is that it gives you a data-adaptive stride for a delta filter.

On some files, you can get huge compression wins by running the right delta filter. But the ideal delta distance is data-dependent (*). The sort of magic thing that works out is that the LZ match offsets will naturally pick up the structure & word sizes. In a file of 32-byte structs made of DWORDs, you'll get offsets of 4,8,12,32,etc. So you then take that offset and forming the LAM sub is just a way of doing a delta with that deduced stride. On DWORD or F32 data, you tend to get a lot of offset=4, so LAM tends to just be doing delta from the previous word (note of course this bytewise delta, not a proper dword delta).

(* = this is a huge thing that someone needs to work on; automatic detection of delta filters for arbitrary data; deltas could be byte,word,dword, other, from immediate neighbors or from struct/row strides, etc. In a compression world where we are fighting over 1% gains, this can be a 10-20% jump.)

Experimentally we have observed that LAMs are very rapidly changing. They benefit greatly from very quickly adapting models. They like geometric adaptation rates (more recent events are much more important). They cannot be modeled with large contexts (without very sophisticated handling of sparsity and fast adaptation), they need small contexts to get lots of events and statistical density. They seem to benefit greatly from modeling in groups (eg. bitwise or nibblewise or other), so that events on one symbol also affect other probabilities for faster group learning. Many of these observations are similar for post-BWT data. LAM sub literals does seem to behave like post-BWT data to some extent, and similar principles of modeling apply.

So, for example, just coding an 8-bit symbol using the 8-bit lolit as context is a no-go. In theory this would give you full modeling of the effects of lolit on the current symbol. In practice it dilutes your statistics way too much. (in theory you could do some kind of one-count boosts other counts thing (or a secondary coding table ala PPMZ SEE), but in practice that's a mess). Also as noted previously, if you have the full 8-bit context, then whether you code symbol raw or xor or sub is irrelevant, but if you do not have the full context then it does change things.

Related posts :

cbloom rants 08-20-10 - Deobfuscating LZMA
cbloom rants 09-14-10 - A small note on structured data
cbloom rants 03-10-13 - Two LZ Notes
cbloom rants 06-12-14 - Some LZMA Notes
cbloom rants 06-16-14 - Rep0 Exclusion in LZMA-like coders
cbloom rants 03-15-15 - LZ Literal Correlation Images

Gamasutra Feature Articles

How to use Photoshop Layer Styles: An advanced guide for game illustrators

May 22, 2015 08:08 AM

"If you're like many Photoshop users, you might not use Layer Styles much. Or maybe you think that they're only good for cheesy bevel and drop shadow effects. Actually, Layer Styles can be very powerful." ...

Desura is not paying developers, but promises a fix

May 22, 2015 12:10 AM

After devs complain publicly about not being paid, the digital download platform releases a statement saying it's "working to correct" problems. Gamasutra speaks to its head of dev relations to find out more. ...

Gamasutra Feature Articles

How Nintendo prototypes games, starring its first new IP in years: Splatoon

May 21, 2015 10:53 PM

The road to creating the ink-based shooter -- which releases very soon -- was anything but smooth, a new interview with the developers reveals. ...



Tapping VR's potential for putting you into an experience

May 21, 2015 09:49 PM

"Two years from now, I guarantee you'll be standing in the middle of the Boston Marathon bombing aftermath, instead of watching it on TV." ...

David Perry Blog

I was recently reminded of this Ben Zander Talk… Worth a listen.

by David Perry at May 20, 2015 08:04 PM

I was in the audience and sadly they didn’t capture everything he did, but the guy and the experience he delivered was incredible.


Filed under: TED Conference Tagged: Ben Zander, Benjamin Zander, Classical Music